It seems like your basically asking for validation that this is ok. If your married and you know your wife would not be ok with this happening, then you know what to do.
If she is ok with it, which by the way your wrote your post, it seems she wouldn't be, then your obviously enjoying it and this guy is too...so carry on...

Keep yourself grounded and don't lie to youself about what is happening...thats what is most important