Herter was re-elected governor in 1954, defeating Massachusetts House Minority Leader Robert F. Murphy. He chose not to seek a third term in 1956. On February 21, 1957 Herter was appointed Under Secretary of State for the second term of the Eisenhower administration, and later, when John Foster Dulles became seriously ill, he was appointed Secretary of State, April 22, 1959. Dulles died a month later. Herter received the Medal of Freedom in 1961.
As an unemployed "elder statesman" after the election of 1960, Herter served on various councils and commissions, and was a special representative for trade negotiations, working for both John F. Kennedy and Lyndon Johnson until his death in 1966 in Washington, D.C., at the age of 71. He is buried at the Prospect Hill Cemetery in Millis, Massachusetts.
Secretary Herter was also an active freemason. He was a member of the Grand Lodge of Ancient Free and Accepted Masons of the Commonwealth of Massachusetts.
Christian Herter's lifetime reputation was as an internationalist, especially interested in improving political and economic relations with Europe.