logoalt Hacker News

yorwba • today at 8:47 AM • 1 reply • view on HN

You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the training data to count possible continuations on the fly: https://arxiv.org/abs/2401.17377 That paper uses suffix arrays, but more compact indices are possible: https://arxiv.org/abs/2506.12229

Replies

anthk • today at 2:46 PM

Could that be implemented in Hailo for Perl (it's two commands away from a base install:)

- Install Cpanminus for Perl, some C compiler and sqlite3 just to be sure.

      cpanm -n local::lib 

     cpanm -n Hailo

     ~/perl5/bin/hailo -t corpus.txt -b chatbot.brn

    ~/perl5/bin/hailo -b chatbot.brn
➕ show 1 reply