I don't know about you, Bob, but I often try to be one of the boys, play poker with the boys, have a boys' night out. I don't want to be called Boy by the foreman at the sweatshop where I work, though.
As to, I would just ask the males here present, I don't prefer use of the word males to refer to men, boys or other human beings. Sounds like Männchen.
Back to the possible context here: I wouldn't mind if the right person said to me, "Be a good boy and open another bottle of Champagne." (Rather less so if, "... and take out the garbage.")
Back to the OP: Depending on context and specifically the relationship, another possible retort would be to use "bub", as in "but it didn't seem to bother you much, though, did it, bub?" or "What's it to you, bub?" Not very friendly, though.