Pretty much what has happened is that you sexualized the event (jerking off) with your friend. I did the same thing with a neighbor growing up. He is very straight, I'm not so straight. It was a bonding activity as you really can't get much more intimate with a male friend. If he is truly the only one that you think about blowing, it's probably only as a way to get closer to him (in your mind). It's amazing how attached you can become to something when it starts in your earlier years.