Japan signed a 1986 moratorium on whaling, but has continued to hunt up to 850 minke whales in the icy waters of the Southern Ocean each year.
The ICJ's ruling is final and there will be no appeal.
A spokesman for the Japanese government said the court's decision was "deeply disappointing".
"We are deeply disappointed and regret that the court ruled that Japan's research whaling program in the Antarctic did not fall within the special permit clause of the treaty," spokesman Nori Shikata told ABC NewsRadio.
"However Japan will abide by the judgement of the court that places a great importance on the international legal order and the rule of law.