there is a lot of solitary imagery in it eg 'i walked across this empty land' which seems like he's alone in the relationship, or they have broken up. Then with the bit about the branches looking at him, it seems as if the 'un certainty' of where it went wron is making him almost..mad/crazy.
The only way to move on is to go 'somewhere only we know' as if that palce where he shared memories of the girl, can show her what they had. May not even be a literal place..but mroe a state of mind...i dont know.