QAM can absolutely be used in fiber optics communication. Encoding and decoding systems are complicated (I won’t pretend to understand it), but it exists.

Example of a research paper using it: https://www.researchgate.net/publication/290224766_Investiga...