Not sure if you tried that already, but ctranslate2 can run BART and MarianNMT models quite efficiently, also without GPUs.
It does! I do use CT2!
On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user.
But it’s definitely worth revisiting that. Thanks!
alt Hacker News
It does! I do use CT2!
On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user.
But it’s definitely worth revisiting that. Thanks!