The 180 rule and the line helps the audience make a mental mindmap of where the characters are in the scene. I think it would be okay just for him to look away and you can definitely break the rule, but only if you're going to focus on what is being looked at. Otherwise I don't advise breaking the rule, cutting from a shot where the line is established to a shot where the line is broken is quite jarring to the audience, it's better to have a cut away or a shot of what is being looked at so that you actually end the line and look at something else and then cut back, instead of crossing it.