If you have enough USB ports, you should be able to get away with using DisplayLink dongles for the video part. DisplayLink doesn't require any additional video processing hardware (unlike DisplayPort) or even USB-C ports, you can run the old DisplayLink standard over plain old USB 2! Not great for playing video, but a terminal app should run just fine.

I think the current driver restricts you to 4 displays but I'm not entirely sure why (you can do 6 on Windows so I think it must be a driver issue).

Of course, with your RPi idea, you could connect one computer to dozens or maybe hundreds of serial ports, but if all you want is five more screens, USB should do fine for your needs.