James was born on Christmas Day 1939 in Marshall, Missouri. His parents were Albert Lamkin "Buddy" James and Alice McElhiney. He also has an older sister named Katherine (Katie). James started playing the piano at the early age of 4. His first piano teacher, Sister Mary Elizabeth, who taught at Mercy Academy, a local Catholic school, discovered that he had perfect pitch. At age seven, James began to study with Mrs. R. T. Dufford, a teacher at Missouri Valley College. He credited her as "an excellent teacher who also taught fundamentals of theory and harmony." His first professional music job was when he was eight years old and playing for a tap dance class at Mercy Academy. "As I recall, I was paid 25 cents, but was eventually let go because my inability to keep the beat over the noise of the tapping.