@MB of RAM would be ok with Illustrator but four would be better.
How do you have your hard drive set up is it partitioned? Even though it is new do you have a lot of files that have been transferred to the new iMac?
How much space do you have on the scratch disk, whether it is the startup disk or a partition?
It does however sound like you have a very large fie caused by the situation Gary describe.
As for you second part of the question I would say the operation of trying to export the file as a .psd of a large though unintentional file has flooded the cache and the problem will go away as soon as you delete the end points. But I would before relaunching Illustrator, trash the Adobe Illustrator Settings Folder in your user's preferences.