I was born in Seoul but moved to London at a young age. I've spent much more time in the UK than I have in Korea.
Being a keen reader however, I have been fortunate enough to have access to books in Korean and to retain my Korean skills. Most people can't tell I've lived abroad. I work in research, which involves conversational, business and academic Korean. I also professionally translate/interpret.
I find my symmetric bilingualism actually surprises more people in Korea than be taken for granted. With people who know about my background, if they hear me speak Korean first, they assume I only went for a short length of time to the UK. If they hear me speak English, they assume my Korean will be impaired, and often make efforts to speak to me in English or to simplify their Korean a bit. Call it condescending, but I like to think it as good will.
It's true that there's an idea of how being born Korean makes you forever Korean in the eyes of another Korean. Not being able to speak the language will make you stand out. Wherever you go, until you start saying something, people will assume that you are just one of 'us', and will speak to you in 'our' language. They may also initially stare when you speak in a foreign language. But I highly doubt that people will judge you upon realising your Korean isn't perfect. Explain that you've never lived in Korea. They may ask questions (main one being 'didn't your parents teach you?') but they won't blame you.
Korea has the highest number of students with international exposure per capita, and a staggering number of gyopo (expats) in general. Second or third generation Koreans who have been born and bred abroad are not alien beings in Korea, especially with so many coming over to soul-search or teach English. So chances are, people have seen (or at least heard about) 'bananas (yellow on the outside, white on the inside)' before. Also, with more tourists coming to Seoul than ever, the bulk of them being Chinese, it's really not uncommon these days to see East Asians around Seoul who speak in their own languages.
If you are worried about fitting in, then I would say the best thing you can do is to be enthusiastic and accepting about Korean culture. Patriotism runs high in this country and especially upon discovering you don't speak much Korean, people will try to teach you helpful words or phrases or to inform you about Korean culture or history. Possibly to the degree that it's annoying. Try and keep an open mind and learn, and to show enthusiasm. I think sincerity and eagerness win over most people.