Just because you're speaking to someone in English doesn't mean that you have to try out every word and every grammatical structure you know.
When you're in a conversation, your first job is just to listen to the other person. If you're listening carefully, you won't have time to worry about your own language problems.
Your next job is to respond to what other people are saying. Your response can be a smile or frown, a few simple words, or a short question. It doesn't have to be complicated.
Don't feel like you have to express every idea that you have in English. Some of your ideas might be too complicated for you to talk about right now. It's OK to save them for later and say something a little simpler instead.