There is a hugh difference between the kinds of challenges faced, depending on which group you are considering.
Native speakers of English, depending on their grade level, need to consider the huge vocabulary that is available in the English language and learn how to correctly use all those words to accurately convey the meaning of their message. It's a wonderful challenge for those of us who love using specific, colorful terminology!
ESL students need to focus more narrowly on learning basic vocabulary and understanding how to handle verb tense changes, pronoun usage, and all the spelling oddities of the language.