imagine how they felt on that exact moment when they heard that luffy's brother died. I mean, luffy basically helped all of them before when they needed his helped, but when luffy needed their help they were no where to be found. They must have been feeling quit guilty.