Upgrading both to level 3 will net you +5 Food and +4 happiness. Upgrading both to level 4 will net you +3 Food and +4 happiness. That's -2 Food after upgrading!!! And I am using the best Roman temple (meaning best happiness rating) while most others give half that much happiness or less.
Sure, you get more taxes if you upgrade to 4, but it is negligible. If your tax rate is 30% (you won't be that lucky considering the skyrocketing corruption), the difference between lvl3 and lvl4 farms is 30.
30 per turn which is nothing. It costs 2300 to build a lvl4 farm. Assuming no extra Agriculture Wealth Modifier, it will take you 2300/30 = 76 turns to get a full return on your investment!!! Stick to Level 3, save the money from the upgrade, and you get 2 more food at level 3.
While the +16 happiness is really really nice, but the -12 Food is HUGE. you don't really need it in most provinces if you build two @level 3 or build one @lvl3 and a Gladiator Circus @lvl3.
Plan ahead, know which plots you really need to upgrade to lvl4 (only a single Barracks in every 2~3 neighboring provinces, the Circus/Race Course in every province for the huge happiness, etc), and keep all other plots at lvl2 or lvl3.
That is of course, considering everything else being equal. Here my example is too simple. I am comparing only two buildings while you should be taking into consideration a balance between all of your buildings in all of your provinces. My advice is don't rush into upgrading buildings once they become available. Do some basic number crunching and see if that upgrade it really worth it.