At some point in your English speaking career you are going to speak to someone who, no matter how hard you try just cant seem to understand you. Due to the large number of English speakers in the world, there are a wide range of accents, some of which are hard to understand. I for one have a hard time understanding some Australian accents and most Irish accents.
You’ll have this problem a lot when talking with people who aren’t used to dealing with foreigners. If they don’t have this experience they will be used to hearing English spoken in a very specific way, and they won’t have any frame of reference when talking to you. Realize that this isn’t a reflection of your language skills, but rather a reflection of this person’s lack of exposure to different people.