The song tells the story of a young man named Tommy, who earns a notorious reputation as the "coward of the county" (and is nicknamed "Yellow," a slang term for a cowardly person) since he never stood up for himself one single time to prove the county wrong.
Tommy's non-confrontational attitude was influenced, at age 10, by his final visit with his imprisoned father, shortly before he dies there (accompanied by the singer, portraying Tommy's uncle). In his final words to Tommy, his father tells him that to "turn the other cheek" isn't altogether a sign of weakness, and implores him to promise "not to do the things I've done; walk away from trouble if you can" (implying that not "turning the other cheek" was what may have landed Tommy's father in prison).
Despite his cowardly reputation, Tommy falls in love with Becky, a local girl who loves Tommy for who he is without having to prove to her that he was a man. One day, while Tommy was working, the three Gatlin brothers (Tommy's bullies, one of whom also loves Becky) came to Becky's house, attacked and "took turns at Becky," in retaliation for her choosing Tommy over the Gatlin boy who loves her. Tommy returns home and finds Becky crying and her dress torn. Reaching above the fireplace and taking down his daddy's picture, he faces the dilemma of choosing between upholding his father's plea of "walking away from trouble", or achieving justice for Becky.
Tommy chooses to visit the bar where the Gatlin boys are. Amid laughter upon Tommy's entrance, and after "one of them got up and met him halfway 'cross the floor," Tommy turns around, this time not to walk away like a coward, but to lock the door behind him ("you coulda heard a pin drop when Tommy stopped and locked the door") and trap the Gatlin boys inside with him. Fueled by "twenty years of crawlin'" that "was bottled up inside him," Tommy engages in a relentless barroom brawl that leaves all three Gatlin boys unconscious on the barroom floor.
Tommy then reflects on his dead father's plea, addressing him respectfully that while he did his best to avoid trouble, he hopes he understands that "sometimes you gotta fight when you're a man.