well, would you be ok with him second guessing you because maybe your boobs aren't big enough? Do you think that your person is worth more than that? Do you think that despite your breasts, you have other things to offer in a relationship?
Maybe thinking about your situation in this light will help you decide.