The phrase appears in print in 1825, in Charles Westmacott's The English Spy:
"With their bunch of fives."
A bunch of fivesIt is also reported as appearing, slightly earlier, in Boxiana by Pierce Egan, 1821. Boxiana is a classic work on boxing history and folklore, and just the place to look for such a reference. It isn't easy to obtain copies of it these days, so that hasn't been confirmed here as yet.