The family remained there for four years. Houdini remembered that time as the happiest years of his life. Appleton was a small market town surrounded by farms. The family lived over a shop on the main street. From their windows, they could watch the townspeople and farmers below. But the apartment was small, so most of their tree time was spent playing outside in the park or in the fields.