“Added you back” means that they essentially “followed you back” if you think of Snapchat like Twitter.
Just like twitter, there’s a few things you’re going to want to be conscious of:
Check your privacy settings. If you don’t want anyone in the world to be able to follow you, make sure you’re not on public.
If you want people to be able to add you, but not be able to see your story, you can do the custom privacy setting and prevent individual users from seeing your story.
Hope this helps. And as far as his snap score, that could be either his one of his privacy settings, or he removed you right after adding you.