At the center of the garden stands the Iso Residence. The residence was originally built in 1658 along with the rest of the garden, but the current building mostly dates back to a reconstruction of the mid 1880s. After the end of the feudal age, the Iso Residence became the main residence of the Shimazu family, and its rooms are preserved in the way they were used in the 1890s. The interior of the residence can only be seen on a paid tour that includes tea and a snack at the end.