The clear allusion in this phrase is to climbing trees. All of us must remember that feeling of not wanting to go further out to reach that apple/ball or whatever for fear that the branch (limb) would break under us. The first uses of it in a figurative sense, with no reference to actual trees or climbing, come from the USA at the end of the 19th century; for example, the Steubenville Daily Herald, October 1895: