alt Hacker News> The fan downloads its root CA cert over plain HTTP. It was always MitM-able.
The most fascinating part of this wave of LLM reverse engineering is seeing how many weird and poor choices were lying in plain sight.
Any number of people could have discovered and exploited this, but it’s tedious work. Most people who can do this are busy with their job and other things. Now LLMs enable people who understand these things to quickly get through the tedious parts, so they can do a lot more of this work in their free time than they could before.
Replies
"The fan downloads its root CA cert over plain HTTP"
Would love to understand the decision-making behind this at Coway (or their firmware contractor). Anyone care to guess why someone would make this choice?
Similarly, how easy things are for more people to do. I'm certainly not advocating for pulling in a root CA over HTTP, but before LLMs could make this knowledge trivial, how many people would even think to look at that angle. It's democratizing in an interesting way.
It is an interesting thought experiment: what things happen once the inertia cost to seeing how something works collapses to 0 for even a small number of your user base. Smart companies might give in now and open up their app even if only as a read-only repo. Dumb ones won’t even know to consider it. And the huge middle will wind up getting new UX ideas for free.
I remember when people were like LLMs write insecure code. Well, so does literally everyone apparently, Maybe that's where it learned it. I feel like that group has gotten a lot quieter lately.
But in some weird way we're coming to the day of reckoning. I'm sure we'll always have missed bugs but with a lot of what we're seeing it more seems like these were avoidable problems. Code issues I can easily imagine at least one person on the team said "this isn't acceptable, we need to correct it" and them getting overruled. I'm sure each and every one of you has seen, if not experienced, this.
It'll be interesting to see if that continues. I suspect companies will still rush but from my experience even the best models happily produce shoddy code. And there's a feedback loop, the model's performance is influenced by the code's quality. So it's easy to degrade and get into a bad feedback loop. But I think people still value speed over quality, even in this world where Microsoft shit the bed
[0] in quotes because the entire conversation is essentially about what is actually good enough and what actually is the minimum viable product.