I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.
NAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.
FPGA model storage?
The basis of Taalas is "compute in memory" electronics - past Von Neumann's separation of processor and memory.
You need to be able to add|mul where the data (the weights) are stored.