Avoid using blame language. Although you don't want to say "it isn't you, it's me" type language, you do want to make it clear that this is about your feelings and your peace of mind. Avoid insulting the character of the other person or blaming them for your feelings.
Remind the other person of the other friends in their life if this seems appropriate.