alt Hacker NewsCallum here, I was the developer that first discovered and reported the litellm vulnerability on Tuesday. I’m sharing the transcript of what it was like figuring out what was going on in real time, unedited with only minor redactions.
I didn’t need to recount my thought process after the fact. It’s the very same ones I wrote down to help Claude figure out what was happening.
I’m an ML engineer by trade, so having Claude walk me through exactly who to contact and a step by step guide of time-critical actions felt like a game-changer for non-security researchers.
I'm curious whether the security community thinks more non-specialists finding and reporting vulnerabilities like this is a net positive or a headache?
Replies
Looks like we discovered it at essentially the same time, and in essentially the same way. If the pth file didn't trigger a fork-bomb like behavior, this might have stayed undiscoverd for quite a bit longer.
Good thinking on asking Claude to walk you through on who to contact. I had no idea how to contact anyone related to PyPI, so I started by shooting an email to the maintainers and posting it on Hacker News.
While I'm not part of the security community, I think everyone who finds something like this, should be able to report it. There is no point in gatekeeping the reporting of serious security vulnerabilities.
Good write up…
I’ve found Claude in particular to be very good at this sort of thing. As for whether it’s a good thing, I’d say it’s a net positive - your own reporting of this probably saved a bigger issue!
We wrote up the why/what happened on our blog twice… the second based on the LiteLLM issue:
https://grith.ai/blog/litellm-compromised-trivy-attack-chain
As a sometimes peripheral and sometimes primary program manager for vulnerability disclosure, for companies you nearly can't avoid, $0.02 follows.
It's a signal vs noise thing. Most of the grief is caused by bottom feeders shoveling anything they can squint at and call a vulnerability and asking for money. Maybe once a month someone would run a free tool and blindly send snippets of the output promising the rest in exchange for payment. Or emailing the CFO and the General Counsel after being politely reminded to come back with high quality information, and then ignored until they do.
Your report on the other hand was high quality. I read all the reports that came my way, and good ones were fast tracked for fixes. I'd fix or mitigate them immediately if I had a way to do so without stopping business, and I'd go to the CISO, CTO, and the corresponding engineering manager if it mattered enough for immediate response.
I've heard stories lately of open source projects being inundated with vulnerability reports and PRs. But in this case, it seems like AI assistance was clearly a boon for root-causing and reporting this so quickly.
Not a security researcher, but this is IMHO obviously positive that the other side of the arms race is also getting stronger, and I would argue it's stronger than on the bad guys' side, due to the best being somewhat responsible and adding guardrails.
I like the presentation <3.
Fantastic write-up and thanks for sharing! I'm sure we will continue to see more of these types of deep supply chain vulns. I think this is valuable for the security community. Remember that Cliff Stoll was an astrophysicist turned sysadmin for Lawrence Berkeley Labs who chased down a $0.75 accounting discrepancy to identify a foreign espionage operation.
thanks for raising the alarm and sharing this, very insightful
(also beautifully presented!)
As someone who works in security, it's really neat that you were able to discover this with the help of Claude. That being said the "I just opened Cursor again which triggered the malicious package" message is a bit eye opening. Ideally the instant you suspected malware that machine should have been quarantined and your security personnel contacted.