I know this for a fact, because when I was growing up, my mother’s “limited” English limited my perception of her. I was ashamed of her English. I believed that her English reflected the quality of what she had to say. That is, because she expressed them imperfectly her thoughts were imperfect. And 1 had plenty of evidence to support me: the fact that people in department stores, at banks, and at restaurants did not take her seriously, did not give her good service, pretended not to understand her, or even acted as if they did not hear her.
If even a daughter can think that her mother is limited because of her limited English, you can be sure that other people think this way also. So you should start improving your accent.