I think as the story progresses, we see that the two are isolated. She wants to visit the Metropolitan Museum of Art and skate and see a French film and “have a nice Sunday.” He makes no objection, but for other reasons. Michael is his infatuation with other women, exploring the “other” or unfamiliar. Therefore, he stares, and this drives Frances crazy. It’s obvious even to Michael that his staring habit is hurtful to his wife.