Think this dramatically simplifies the problem. AI existed before GPTs and the AI in self-driving is optimized for self-driving and the latency you already mentioned.
Regardless of how the AI is architected, you aren't going to be able to use a generic LLM like Qwen to perform reliable self-driving, you need a highly optimized, highly specific AI.