alt Hacker News> There was this really interesting one where it reads your lips
This is what I thought the original post was about. The glasses you have to wear is where that lost me.
If we could get lip-reading via your computer/phone camera down perfectly, that's the real win. Making people wear your goofy frames is where you lose a massive portion of people. Nobody wants to look like Zuckerberg lol
Lip reading is one of the other modalities we tried as well beyond IMU, EMG and RF! We worked a lot on fine tuning Auto-AVSR, one of the most commonly used open source lipreading models. It's very enticing to work on, since it gets to about 17% word error rate, which looks good enough that you would think working on it further would get it down to product quality, but we found there's an information-theoretic limit on the modality. Speech has 40 phonemes, but only about 12 to 15 visemes (externally visually separable phonetic states). For example P and B look very similar from a camera. This causes the scaling laws on lipreading to flatten out at an error that's too high to be product ready but just low enough to make for a good demo. Our original plan was actually to ship a lipreading software to see if people would want this behavior, but we realized you couldn't make lipreading work!