It may have been that all of the Orcs with even the slightest bit of wisdom had been present at the mine during our attack. As such, all of their supplies, leaders, Mages, and organizational hierarchy were all wiped out. The survivors are probably no more than mere beasts that rely on pure instinct. A better way to picture what is going on is to imagine all of the survivors are like Ogakichi-kun without my influence. With their food running short, they are attacking the closest source of food possible, meaning the town and those that went out to collect the lumber.