Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc.

Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density