Eisenhower named Herter undersecretary of state on February 21, 1957, from which post Herter succeeded Secretary of State Dulles in 1959. While not as close to Eisenhower as Dulles, Herter was an important figure in the administration; much of his time was spent dealing with the U-2 spy plane incident in 1960.
After leaving office, he served as cochairman of the United States Citizens Commission on NATO in 1961, urging closer cooperation with the European countries. President Kennedy named him the President's Special Representative for Trade Negotiations, on February 27, 1962, and he continued to serve in that capacity under President Johnson until his death in 1966.