alt Hacker NewsDeepSeek V4 hallucinates like crazy and often forgets explicitly mentioned parts of the context. I guess compressing tokens and cherry-picking attention comes at a cost.
Replies
Deepseek V4 pro is a heavily undertrained model, they only trained it a bit more than the small version and that small version is 6-ish times smaller. Ive found that Flash is absolutely incredible as a workhorse for wide scale agentic nonsense, but Pro is a bit undercooked and really goes on strange tangents very often.
I have seen occasional weird behavior that I guess could be attributed to hallucinations, but for security auditing, DeepSeek v4 Pro is among the best models I've tested, competitive with Opus 4.8 and GPT 5.5 (MiMo and GLM also did well, Qwen 3.7 Max was below all of those, though only barely), and at an order of magnitude lower cost per task.
Deepseek is one of the worst in terms of hallucination rate according to artificial analysis' benchmark: https://artificialanalysis.ai/?omniscience=omniscience-hallu...