You should not care what others think. They may not think the way you imagine negatively. They might even admire you or be envious of you. You cannot see through their minds. It is only your imagination.
For the past 10 years, I have spoken English at home with my 2 grown-up children. I also speak English when we go shopping at Central near our home. In fact, I hardly speak Thai with my two children whether at home or anywhere. I never care about others' opinions whether positive, neutral or negative. I care about providing the opportunity for my children to speak.
Remember: Use it, or lose it.