Baker was born and raised in a musical household in Yale, Oklahoma; his father, Chesney Baker, Sr., was a professional guitar player, and his mother, Vera (née Moser) was a talented pianist who worked in a perfume factory. His maternal grandmother, Randi Moser, was Norwegian.[5] Baker began his musical career singing in a church choir. His father introduced him to brass instruments with a trombone, which was replaced with a trumpet when the trombone proved too large.
Baker received some musical education at Glendale Junior High School, but left school at age 16 in 1946 to join the United States Army. He was posted to Berlin, where he joined the 298th Army band. After leaving the army in 1948, he studied theory and harmony at El Camino College in Los Angeles. He dropped out in his second year, however, re-enlisting in the army in 1950. Baker became a member of the Sixth Army Band at the Presidio in San Francisco, but was soon spending time in San Francisco jazz clubs such as Bop City and the Black Hawk. Baker once again obtained a discharge from the army to pursue a career as a professional musician.