One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).

A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).

Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.