Once he left, he couldn’t ensure that these two fellows would not secretly harm Fourth Mother.
If Fourth Mother did not leave, he too would absolutely not leave.
Moreover, if by any chance, the battle against the swordswoman and Xue Tan Lang ended up with the three of them becoming seriously injured, he couldn’t be sure that he would not be taken advantage of by First Uncle Yue Shan and Second Uncle Yue Ling. Yue Yang decided that it was more important to preserve his life than fighting and killing.