A person has what is known as 'character' and 'personality'. These words indicate discenable patterns of behavior. Determinism in this discussion means that a given person will 'behave in an expected manner'. Further feature of humans in that you can coerce them to behave in highly constrained manners.
Entire institutions and organizations in humanity are based on this feature of human beings.
A heuristic software agent that will respond in a spectrum of possible ways and past performance is no guarantee of future performance. It is no way reassuring that your agent has 'never done x' since if 'x is in the space of possible responses' it can certainly do 'x'.
> Determinism in this discussion means that a given person will 'behave in an expected manner'.
Oh I see. That is not what determinism means. Determinism means their actions are predestined, "determined" by the initial conditions. Computer programs - including AI - are deterministic, barring cosmic ray flips. The same inputs running in the same environment produces the same output.
Humans are or are not, depending on whether free will exists.
Either way, it's hard to draw the boundary around determinism such that it includes humans but excludes AI.