It was considered legal for a Brahmin to physically abuse a śudra (but it was a crime for a śudra to abuse a Brahmin). For a Brahmin to kill a śudra was supposedly no more serious than killing a bird.
Essentially, the śudras were deemed to be predetermined by their caste status to be slaves.
During the Vedic period, śudras were increasingly excluded from public rituals, such as the higher-caste initiation ceremony (bestowing the sacred string on a son).