Prime Minister Shinzo Abe said his government would "start over from zero" and find a new design.
The original design, by British architect Zaha Hadid, had come under criticism as estimated building costs rose to $2bn (£1.3bn)
Mr Abe says the new stadium will still be completed in time for the games.
However, the delay means that the stadium will no longer be ready in time for the 2019 Rugby World Cup, which Japan is also hosting.
"I have been listening to the voices of the people and the athletes for about a month now, thinking about the possibility of a review," Mr Abe said.