I wonder how this behaves when quantized, and how far are we from being able to run something like this purely in the browser with WebGPU.

If you swap the text encoder for something like T5-small or CLIP, this could be <2.5Gb at INT4; doable on a lot of machines.