alt Hacker NewsSince most folks still somehow review the code, I find it highly improbable, it would become visible quickly. Unless say compiling Unix parts for example. But thats compiler work and not llm.
Replies
majorchord • yesterday at 4:49 PM
Jia Tan would disagree.
The overwhelmingly vast majority of open-source code isn't actually looked at or audited. Yes it's there for all to see, but that doesn't mean it's doing any good at the moment, in this context.
Its not the code so much as the model. The model itself can be poison pilled and embedded with ideas that make it unfit for use and there isn't a person out there who can look at the weights and tell you what or where the problem is by looking at the code.
this isn't if-else statements, its a jumble of linear algebra and matrices.
I watched a demonstration where one AI was trained to be obsessed with penguins. they asked for a random set of numbers from it. They fed that set into another AI model to analyze and the new model started to become obsessed with penguins.
I dont think open source is the only one to worry about though. I don't know if we even have a way to guarantee any model is secure, open or closed.