In retirement after his second term, Madison managed Montpelier but continued to be active in public affairs. He devoted long hours to editing his journal of the Constitutional Convention, which the government was to publish 4 years after his death. He served as co-chairman of the Virginia constitutional convention of 1829-30 and as rector of the University of Virginia during the period 1826-36. Writing newspaper articles defending the administration of Monroe, he also acted as his foreign policy adviser.
Madison spoke out, too, against the emerging sectional controversy that threatened the existence of the Union. Although a slaveholder all his life, he was active during his later years in the American Colonization Society, whose mission was the resettlement of slaves in Africa.
Madison died at the age of 85 in 1836, survived by his wife and stepson.