The answer to your question is yes... I've had this problem (I've had four lovers and my size has always been an issue). And while you can do a lot of things to help it (more foreplay, lots of lube, etc.), the fact remains is that we don't all fit together perfectly.