Technically "fantasy" is something that exists beyond the realm of reality or even potential reality. That's why Star Wars, for instance, is definitely fantasy and NOT science fiction. The way that people use the term to describe their sexual desires, though, isn't usually the same. So it depends on the person.

Some people may say they have a fantasy about having a threesome, and they may very well want to have a threesome and may very well have one at some point in their lives. Other people might have a fantasy about being raped, which is pretty common, but they would never actually want this to happen in the real world, even though it realistically could.

The most accurate use of the term would be to describe something that could never realistically happen. Such as having a fantasy about having sex as a member of the opposite gender, or having sex with Cleopatra or some other long dead historical figure.