Thought this was gonna be about CARDIAC, lol.
Emacs, and technologies built on it, such as org-mode, come somewhat close to ideas expressed here by having plain text in a buffer be the unifying data format. You can organize stuff by just moving snippets of text around.
I think it's difficult in practice to design data manipulation interfaces based on real-world objects because atoms are heavy and bits are not. Data is just much more malleable and transformable than real world objects, at least at the pre-Diamond Age tech level we're at. But maybe ML will help make this easier by allowing computers to track and scan the objects more easily.
https://www.instructables.com/CARDIAC-CARDboard-Illustrative...
Although the cardboard implementation is kind of the point, I think it's cool that someone made an FPGA version (dead link though, RIP drdobbs.com).