A visit to Kyoto is a must. If you don’t go to Kyoto, and for example, only visit Osaka then do check out Osaka Castle, or whatever the major castle of the area or city is.
Temples and shrines are everywhere and are never far off the main tourist track.
If you travel to Japan in their spring (around April) then you must see the cherry blossoms, or go to a cherry blossom festival. If you do not travel to Japan in spring, then find out what festivals are on and go see them, you will experience great local Japanese food and culture.