Since they cannot see the expressions on his face, the people fear what the minister sees, and, above all, what he may know about them. Mr. Hooper's wearing of the black veil causes the congregation consternation about their own hidden sins, and they fear discovery. They also worry about what sins of his own that he may be hiding.