I don't know, it's probably a little bit weird. I call a couple male friends my buddies, but I don't think they refer to me the same way. If I hear a dude say "buddy" I definitely think male.
If you add "drinking" or "fuck" before it though, then it works a little more.