You can always ignore the BIOS routines and directly touch hardware registers/etc. so as long as you know precisely what hardware you are dealing with. Of course, this is what modern OSes do for most hardware after bootstrapping since calling back into BIOS isn't really an option.
BIOS routines are purely an abstraction layer, though the abstraction is somewhat leaky and my understanding is that most hardware was trying to be IBM compatible even when software skips the BIOS routines and directly touches hardware interfaces.
e.g. you can see documentation for the VGA card here: https://wiki.osdev.org/VGA_Hardware
The thing is, as far as I know you do not need to use the video BIOS routines even to set the video mode. After all, the video BIOS routines are also just routines that run on the CPU, the only advantage they really have is that of any abstraction, the fact that using it allows you to be compatible with any card that implements that software interface even if it doesn't implement the same exact hardware interface. But as far as I know, if you know you're dealing with a VGA compatible card, you can set the mode by directly flipping around CRTC registers and it should work just fine.
Same for disk controllers and etc.
However, since GPUs tended to vary widely when going beyond VGA modes (although I believe even the original VGA controller was capable of 800x600x4 with a suitably lenient autosync monitor), VESA VBE was introduced to make that much easier again.
Windows actually runs the VBIOS in an emulator / VM for switching modes with the default GPU driver.