WebLLM: high-performance in-browser LLM inference engine

This seems to be the demo:

https://chat.webllm.ai/

I am getting:

    WebGPUNotAvailableError: WebGPU is not supported in
    your current environment, but it is necessary to
    run the WebLLM engine.

It is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.

I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.

Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.

You might like webml-kit https://npm.im/webml-kit

Bake me a cake

responds with

> Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.

A WebX technology that actually involves browsers!

[flagged]

[flagged]

[dead]

[dead]