logoalt Hacker News

pplonski86today at 6:35 AM1 replyview on HN

Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.


Replies

chiitoday at 7:00 AM

> I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth

what model was it that you were able to run with the rtx 3070?

show 1 reply