Hank Palmer (Robert Downey Jr.) is a successful defense attorney in Chicago who has been estranged from his family in a small town in Indiana for a couple of years. His brother Glen (Vincent D'Onofrio) calls with news that their mother has died. As he leaves to fly to his hometown of Carlinville, it is revealed that Hank is going through a divorce and custody battle for his young daughter.
Hank reunites with his brothers: older brother Glen, an owner of a tire shop, and younger brother Dale (Jeremy Strong), an autistic aspiring filmmaker who carries an old 8mm film camera everywhere he goes. Hank then stops by the small county courthouse, softly enters a courtroom and sits in the back where he proceeds to observe the judge presiding over a matter. The judge is Joseph Palmer (Robert Duvall), Hank's father.
While watching the proceeding Hank sees his father forget his bailiff's name (who has been the Judge's bailiff for years). Hank thinks the death of his mother may have driven his father, a recovering alcoholic, to start drinking again. When Joseph sees Hank for the first time in many years at the funeral, he shows distance to his son but treats guests with more kindness while everyone is giving their regards.
After his wife's funeral, Joseph excuses himself to go shopping while the brothers catch up with each other. Joseph rebuffs their numerous offers to accompany him. After Joseph leaves, his sons go to a local tavern to reminisce about how when they were children they would fish at a lakeside cabin with their father. Dale mentions that Joseph still goes to the cabin every week to play chess with his old friend, Dr. Morris.
The next morning Hank notices damage to his father's Cadillac Coupe DeVille as he is preparing to leave. Upon seeing the damage, Joseph denies knowing how it happened, but the brothers know he is lying. Hank believes his father was drinking and driving and damaged his own car. Joseph responds by criticizing Hank's failed marriage. A heated exchange ensues, ending with Hank leaving, promising to never return. While boarding his plane to return home to Chicago, Hank learns that Joseph is being questioned by police with regard to a fatal hit and run, the victim being someone he convicted years earlier.
Joseph is soon indicted, with prosecutor Dwight Dickham (Billy Bob Thornton) seeking a conviction for first-degree murder. Hank ends up defending Joseph and learns that his father was undergoing chemotherapy. Hank also reconnects with his old girlfriend Sam (Vera Farmiga), whom he abandoned after going to a Metallica concert more than 20 years ago. He discovers that Sam has a daughter, Carla (Leighton Meester), born nine months after he left. It is later revealed that Glen, not Hank is Carla's father.
Despite Hank's efforts, Joseph is found guilty of voluntary manslaughter and sentenced to four years in prison, but is released on compassionate parole after having served seven months. He and Hank go fishing. After recognizing Hank as the best lawyer he knows, Joseph dies in the boat. After the funeral, Hank goes to the courthouse alone, and puts his father's hat on the bench. He spins the seat, and when it stops it faces him.