While increasing your Physical Attack with items only gives you higher damage because of the Scaling of the skill, a point in All Attack in the Stat Growth System, is increasing your damage twice. On one hand it increases, as your items, the damage due to the Scaling of the Skill but in addition it does also increase the Base Damage of the Skill.