they probably mean deterministic in the sense of traditional computer programs consisting of if-else decision points and ordinary cpu logic, as opposed to numerical models (which can of course be deterministic too, as you pointed out)

There are 2 levels of determinism that matter here IMO. Level 1, given the same input you get the same output. Level 2, you know what to change on the input to get a specific change on the output.

LLMs are currently capable of level 1, but not capable of level 2. Trust follows what is actually deterministic.

I mean in the traditional software sense. We trust autopilots to fly planes. Would you get on a plane flown by an LLM?