Its output voltage is also affected by current through R4.
When the op-amp's output pulls to 0V, R4 is connected almost in parallel with R1. It's not exactly in parallel, because the op-amp output doesn't pull all the way down, and there is voltage lost across the diode, but it's fairly close.