When the relationship brings you more pain than joy.
Sometimes, we tend to be blinded by the past happy moments of the relationship. To the extent we forget about all the unhappiness it brings us. If your relationship leaves you frustrated/upset/unhappy more often than not; If your relationship is leaving you in tears every so often, perhaps this might not be the right person for you. The relationship you are in now should be one which brings you happiness now. Just like #1, if the main source of happiness of your relationship is from past memories, something is amiss