alt Hacker Newsinteresting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference
interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference
That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.