If you didn't want it, you wouldn't let him do it. As far as the abuse issue, that's a bunch of bs. He found you hot, he sent you a few signals that you seemingly had no problem with, he escalated, you accepted.

There might well come a point where you need to set boundaries. Be prepared to say no if you don't want it to go any further than it has. There's nothing wrong if there's no discussion about it, sometimes that can be really hot in and of itself but you need to be able to say no if he crosses any boundaries.

What you say to your wife is up to you but IMO, as long as it remains casual, then there's no need. A problem could arise when and if you stop getting massages from him. Then, if you start hankering for more of the same and go hunting for it, you need to be more open with yourself and start thinking in terms of safe sex and how your actions might impact your wife.

Until then, enjoy!