If he do something to Aiko and the others when Hajime’s party went to the frontline, the possibility it become a chaos was high. However, because he actually didn’t do anything, he was convinced that he didn’t want to do it directly, without any basis to it.