I think this was true for MDA (text only), but the Hercules had 64k of video memory at B0000, the latter half of which would overlap with the CGA card, and also VGA's text mode.