Remember, make yourself clear, repeat if you need to so that he knows how to get there without being confused.
What if you don't know the correct directions ?
There are times you may not know how to give people directions because you are new in the city/area, or you haven't heard of the place that he wants to reach. Don't be rude by shrugging your shoulders and walking away.
You could always say :
I'm sorry. I'm not from here so I don't know my way around.
I'm afraid. I can't help you.