balls: Regardless of what you thought or how you felt, you were molested. It is the duty of a society to pass judgement on such behavior, that is why we have laws.
Did your parents approve of your relationship with the older man? Did they say, "Sure, go ahead and suck his dick." You make it sound like it was a very normal occurence, it isn't. You were molested, he should have been charged and sent to prison. The scare thing is you were probably not the only one he molested.
By the way, the reason your relationship slowed and probably ended is because you were no longer a little boy.