The genre where we find the "fantasy/sci-fi" books right next to usually mystery books in our German book stores usually is speculative fiction:

- mystery, - horror, - fantasy, - and sci-fi have lots of overlap. - And XKCDs What-If-Books even.

But as someone who hates fantasy and loves sci-fi, the biggest difference for me comes from plausibility: "a witch did it" as Occam's fantasy razor for things being the way they happened, and the underlying physics engine means for me: either this is soft "sci-fi", or they better explain with hard rules the limitations and effects of their magic system.

Since (low-sorcery) fantasy RPGs even have torches that would never work, when historical torched looked completely differently, suspension of disbelieve hardly sets in for me in fantasy books.