Of course physical appearance matters. The person you are meeting for the first time does not know you and your appearance is usually the first clue he or she has to go on.
But it certainly does not mean you need to look like a model to create a strong and positive first impression. (Unless you are interviewing with your local model agency, of course!)