This result can be explained by the fact that glycerol is soluble in alcohol.
Therefore, too excess of methanol can increase the concentration of glycerol in reaction mixture which can shift the equilibrium to the reactant side, and resulting in the decreased FAME yield and oil conversion.