alt Hacker NewsHad a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.