It sounds like he's just being cautious (like you are... and both of you SHOULD be!)... meet him somewhere. Have a cup of coffee. Don't have sex with him. See if the physical connection is as strong as the mental connection has been.

This life is too short not to take a risk on what could potentially be a wonderful friendship or relationship.