The song is about nostalgia. It’s because of nostalgia that he is talking to someone he was once in love with. But, it can be applied to anyone; someone you’ve had some sort of relationship with.
He has a desire for a simple moment where this person takes him somewhere only they know. In order to let him in, and talk about where their lives are going.