In short, basic math deals with numbers. Since we are all taught how to count at a young age the concepts of basic math, even though challenging at first, seem to have a practical value - even to children.
Enter Algebra. Suddenly, we are asked to deal not only with our comfortable numbers but with letters. And it doesn't stop with this. You start seeing parenthesis and exponents, and a whole potpourri of other symbols that seem to make no sense at all. This single fact more than any other turns many people off to learning algebra. At the very beginning you are asked to learn certain rules on how to calculate things in algebra. You must learn which steps are legal to do before others, and if you do them in the reverse order you get the wrong answer!