Tried this out on my ThreadRipper:

AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I assume are pure AI slop (LLMs love to estimate incorrectly. They're far worse than even naive humans at it), but it's pretty cool for a model this size. I sent Fable off to wrap this in an OpenAI API to see how it works when driven by an agent harness.

EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool:

> The meaning of life is one of the oldest and greatest questions in human history, yet strangely, there is no single, universally agreed-upon answer. Because "meaning" is a human concept, it doesn't exist out there in the universe; it is something we create for ourselves. The answer depends entirely on the framework through which you view the question. Here are the most common ways to answer it. The meaning of life is the meaning you give to it. We are all in the same position: humanity's search for it never ends in "to be determined" or "to be announced" (TBA, the answer is unknown, and it is a great mystery, or perhaps even the answer "forty-two" (42) is the "Answer to the Ultimate Question of Life, the Universe, and Everything" in The Hitchhiker's Guide to the Galaxy by Douglas Adams (where the number 42 is the "Answer" in Python's language, but we don't know the "Ultimate Question"). Here is a joke that works under the frame of "A..." (any answer): "A clean desk is a..." (42 is a "portmanteau" of words and just a great big "Ad..." (Ad-100) and "A&d" (100)). Life is a deep and strange and we search for meaning in it. "I think, therefore,..." (Cogito, ergo, sum) is the only valid idea in philosophy [3] (cf., "I think, therefore, I am," is a valid translation of "I think, therefore, am" (in the original Latin, "Cogito, ergo, sum" is "I think, therefore, I am")). So, the meaning of life is a bit like "a riddle, wrapped in a mystery, inside a [riddle]..." (G. K. Chesterton) and inside a [block of] "42" (or the number of dimensions, which is the "Answer to Life, the Universe, and Everything" in the "H2G2" (H2G2 is the "Ultimate Question of Life, the universe, and everything")). The "H2G2" is a "puzzle, wrapped in a mystery, inside an enigma" (cf. [3]). We are all in the same position, but we all have to give it a meaning. our own meaning. The meaning of life is what you make of it. The meaning of life is to live for the greater good. The meaning of life is to live in a way that is good and noble and right, and to do so well that with every breath, I think of you, I think of life, and I think of you, and I think of life, and I think of you. (cf. [3]) If life in the universe is a "great question," the answer is 42. The meaning of life is the meaning you make it. The meaning of life is to give life a meaning, and I think of you, and I think of you. So, the answer to the ultimate question of life, the universe, and everything is: 42. The meaning of life is 42. The meaning of life is the meaning of life. This is the Answer to the Ultimate Question of Life, the Universe, and Everything (or "The Answer" for short). It is the Answer to "the" Ultimate Question of Life, the Universe, and Everything. (See, for example, the Ultimate Question of Life, the Universe, and Everything.) This is the answer to the Ultimate Question. This is the Answer. (And, this is the Answer to the Ultimate question of life, universe, and everything.) The meaning of life is the meaning you give to it. The meaning of life is to give it a meaning. The meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is a list of the possible meanings of the universe of life. It's a list of the most common and accepted answers. "What is the meaning of life?" The answer is 42. The meaning of life is the meaning of life. The answer is 42.) (See also: [3] for a list of possible meanings.) The meaning of life is to give it a meaning, and the meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is the answer to the Ultimate Question of life, the universe, and everything.) (This is the answer to the Ultimate Question.) The meaning of life is 42. The meaning of life is 42. The meaning of life is 42. (See also: [3]) (The answer to the Ultimat