carolinacurious: I wouldn't say anything to him about his our your penis.

You could just ask him if you've done anything to offend him or why it is that he seems to have a problem with you. (depending on his answer maybe then you could say something about the luck of the draw, or how you're a show-er not a grower, or how you're embarrassed by it (even if that's not necessarily true), or the girl's are exaggerating or whatever.

My question is this:

Ok, you want to go do something, the girls want to go do something, he wants to go do something (but not if you are involved); what happens? Do the girls stay with him, go out with him and leave you alone, or do they go with you and leave him to sit and stew?

If the girls go with you then it's simple, he either gets over his problem or he gets to spend some quality time with himself.