Depends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".

> we cannot step debug an llm's output to find out what happened

We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.