Tip #8: Use Resources folder when necessary
The solution for the previous case is place all those prefabs/assets in the Resources folder and load dynamically the one you need. It's not as convenient as references because you will have to provide a path to the asset and moving it to another folder will cause problems. But I guess you can write an Editor script that handles that for you :)
An slow game is a dead game. Players usually accept many degrees of mediocrity: bad user interface, bad story, bad graphics... Everything can be forgiven but bad performance. You need to know your limitations and in situations where you have to decide you should usually sacrifice everything for the sake of decent Frames Per Second. That's the hard truth xD