Why not have some a device/hardware that programs itself on-boot.

Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.