There is a certain logic that supports the teacher’s view.
Older students and those who are better at math don't use their fingers, whereas younger and less-apt students do. One goal of education is to make younger and less apt students more like older and more apt ones.
Therefore, children who use their fingers should be discouraged from doing so.