> The display of [a man page] is performed by the program defined in the MANPAGER or PAGER environment variable, or by a default program, usually less(1).

A man page source isn't a binary format, so your statement that they're "plain text" is technically correct. (The same is also true of TeX and LaTeX files, and even PostScript if you want to stretch the definition of "plain text" until it snaps.) But the renderer is groff or (legacy) troff with the `an` macro set. less(1) (or, originally, more(1)) is just the pager that consumed the former's output (if the output format is ASCII, which is one of many) and handled paging on the terminal for the convenience of the user.

In my old Sun workstation (and even early Linux desktop) days, I rarely used man(1) in the terminal because 1/terminals were usually too small and weren't usefully resizable like they are today, and 2/unadorned monospaced fonts don't look nearly as nice as properly typeset pages do. (Color terminals were just coming on the horizon, and text could only be emboldened, not italicized.) Instead, I typically used xman whenever I could. The best way I can describe xman is as if you were rendering man pages into PDFs and viewing them in Preview on the Mac. Man pages were much more comfortable to read that way.