This is not uncommon. For some men, the testicles never descend to an external position in the scrotum, and that is a problem. For some other men, the testicles are descended into normal position, but can retreat back inside the body (usually from either cold or fright - both can cause retraction). If it occurs often, you should probably mention it to your doctor. It's probably not good for them to spend too much time up there.