One survivor told AP: "I woke up in the middle of the night and the corridor to the living room of my house was already flooded."
"I heard the sound of water coming in, and then the water from the river rushed into my house, so I just took the car and rushed out."
A university professor in Hiroshima, JJ Walsh, told the BBC that everyone was surprised by the scale of the storm.
"Most people here are used to heavy rain. We have a rainy season. But I think everybody was pretty caught off guard by how much rain it was," she said.
Prime Minister Shinzo Abe said he had asked local officials to "raise the number of [military] personnel to several hundred in order to strengthen rescue operations".
More rain could trigger further landslides, weather officials have warned.
Much of central and southern Japan is mountainous, with many homes nestled into steep slopes.
Last year, a typhoon triggered landslides on Izu Oshima island, south of Tokyo, that left 35 people dead.