Damnit Peter, now you nerd sniped me. What ideas were you looking at?

Here's the architecture I'd use. I'm going off of NVIDIA's current recommended stack / what seems most mature, if I understand correctly after a quick chat with GPT.

Basically I'd start with a bunch of Xeon 6s (Granite Rapids) connected to Blackwells. You could do it with AMD too but I trust their stack less. That gives you TDX as a starting point, which is pretty mature by now. I'd prepare a very minimal Linux image in which the OS boots straight into running a training server. You need the usual stuff like dm-verity or IMA to ensure the disk image is covered by the kernel image. The image also contains NVIDIA's driver and attestation SDK. Boot it all inside a TDX VM with GPU passthrough. I believe that currently uses encrypted bounce buffers so there'd be some perf loss, but supposedly there's some PCIe extension that can be used to eliminate that (I haven't investigated).

The trick is to combine NVIDIA's RA with Intel's. The best way to do that is just hash the evidence from the GPU attestation process into an intermediate data structure which also contains the usual stuff (e.g. ECC pubkey, nonce), that itself is hashed into the user slot of the TDX RA and then bundled all up together. After doing this you have a single über-data structure which can be used to prove that the given public key was generated inside a genuine Xeon TDX VM connected to a genuine NVIDIA GPU running in CC mode. I'd probably just use a zip file to collect all these bits of data together, for convenience when implementing the verifier. It might be worth including an RFC 3161 timestamp over all that as well. The RA protocols for something like this are always very tricky: I'd need to do some careful design work before committing to anything so don't over-read this comment.

Now the TDX RA covers the training server (indirectly, via the dm-verity root via the kernel args+hash). What is this server? It's training code (e.g. PyTorch) combined with a server that accepts document batches and - for more advanced training - maybe stuff like AllReduce traffic. Each time a document batch is accepted each doc is hashed into a sorted Merkle tree. The weight updates are then computed on the GPU and the results are hashed into a second accumulator.

Each node creates a blockchain-like log in which each record is signed by the static key embedded into the TDX RA. The records point at the hash of the previous records, and contain the current value of the Merkle root and weight accumulator. A verifier can rebuild the roots by iterating over the training set themselves in the right order and just hashing the text. I won't go into the details as you know how all this stuff works already.

I'm handwaving a lot of details around parallelism here. For a large scale training run with many nodes you'd need to solve a bunch of details like making the training servers verify each other's remote attestations before they join the cluster, etc. During the AllReduce step nodes would need to verify the data comes from a authenticated TDX VMs before accepting the gradients.

Anyway from time to time the weight updates have to be applied and the log also records the hash of the full sets of weights at that checkpoint. The final model consumer can be given the weights, along with the signed logs generated during the training run. They verify the chained RAs, verify the signatures on the logs, and if they wish, verify [a subset of] the hashes of the documents in the open training set.

Because the document Merkle tree of each batch is sorted, you can build both inclusion and exclusion proofs. Verifying an exclusion proof is more expensive as you'd have to check each batch versus a quick Merkle branch check for inclusion, but exclusion checks can be done in parallel. Normally in ML documents are shuffled rather than sorted, but the training server can sort the set for building the log entry Merkle tree, and then submit to the GPU in a shuffled order.

The bigger issue for exclusion proofs is that SHA256 is too rigid. You might need some sort of simhashing scheme to run along side it. Dunno, needs more tightly specified requirements.