Yes, the OP is right. Most of it all is purely PSYCHOLOGICAL. Size does have an effect though.

Hear about a girl who couldn't get off on a "small" cock? It's because of that. Anatomically she's perfectly able to get off on any reasonable sized member.

If a girl goes into sex thinking "this guy has a small penis" its going to be harder for the guy to get her off.