He was born in Osceola, Arkansas, the only child of Kemmons and Ruby "Doll" Wilson. His father was an insurance salesman who died when Kemmons was nine months old. Shortly thereafter, his mother, Doll, moved the two to Memphis, Tennessee, where he was raised solely by her.
Wilson was married to Dorothy Lee. They had five children: Spence, Robert, Kemmons Jr, Betty, and Carole. Wilson died in Memphis aged 90 and is interred there in Forest Hill .