If the crystals are growing in size while the number crystal concentration is constant, the network will get coarser and absorb an increasing amount of liquid phase.
This effect may be further strengthened by the presence of non fat particles that may magnify the impact of a coarsening fat crystal network.
Thus our picture of the processes is that there is an initial phase of molecular diffusion followed by a re-crystallisation processes and crystal growth leading to coarsening network and convective oil migration into the expanding shell matrix.