I will make it more clear by using strong words. (sorry)
So every one will understand.
If there was a man walking in to your house, passing your eyes, to JACK your wife.
and you intentionally act like you did not see that man. (you may want to hide something)
and you are telling to the others that the man was just walking around your house, not passing thru.
Should you blame your neighbor for telling you late ?
It is your house, your eyes, and your wife, Right ?