At the age of 6, Freeman's grandmother died and he moved north to be with his mother, who had already separated from her alcoholic husband. They later moved to Tennessee and eventually back to Mississippi, where Mayme Edna settled her family in Greenwood.