Wouldn't that depend whether or not the wall that the 4 men are building is the same size as the one the 8 built.
Plus if less men are building it, then they may not finish the building in a working day, so either they would have to work overtime, or continue into a second day, which increases the amount of time the building process takes.
Factor into that, with less people building, there is more chance of a work related injury. The inspection to see if the worksite is safe and compliant to the safety code would be stricter, and keeping to the recommendations that the inspectors give is costly and time consuming.