alt Hacker NewsI quit OpenAI because its culture is broken
Comments
How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
Love living in a manifestly dysfunctional country with no regulatory state. Just running the country on autopilot until something catastrophic happens.
Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.
I feel like I'm missing something. They seem to not even have basic observability and online evals. It's not that hard to look at a trace and answer the question "is it writing code for that makes an external network request?" with reasonable accuracy.
I have made enough money working in AI that I can now speak my mind about AI
Trolley problem:
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.
Software systems have bugs. They always do. This includes security bugs: vulnerabilities in code that could be exploited. In any software project I’ve worked on this is always true. We never finish fixing security bugs. That means vulnerabilities are always present even if hard to notice. AI tools are amazing at finding hard to notice bugs. It seems obvious that AI tools will have the capability to escape any sandbox. Ordinarily we design these sandboxes as a countermeasure to human bad actors. But what happens when the bad actor is supercharged? I think it is reasonable to be worried. It’s also reasonable to distrust those building this tech to adequately safeguard. Because how can you really?
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
We risking to loose knowing who is the head. A new "ORGANISM" is being created as we speak but we don´t know how the head looks like. Good luck figuring that out with this terribly disorganized progress. What can go wrong!
PR firms such as Spitfire Strategies got steadily growing stream of business. It’s great for the GDP, diversifies the economy from compute-centric growth.
> this moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.
Exactly, you can't expect exploitive, manipulative executives that came to 'win' in capitalism to stop acting as to their nature - they need to be cleared out and people who think and care wholesomely need to be given the reign so all of humanity succeeds.
Archive link to original Atlantic article: https://archive.ph/5GQx8
This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.
1. https://www.nytimes.com/interactive/2023/11/20/technology/le...
My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job
I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.
I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.
Well this is a great sign for OpenAI. I'm sure the typo inclusive memorandum will save us.
when ppl resign do they give up any equity? just asking because i don't know.
Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?
I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.
A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
> Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.
Yeah so that's never going to happen
Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
Aligned/unaligned is counting angels on the head of the pin.
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
Reminds me of a famous quote from Jurassic Park (1993): Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." - Dr. Ian Malcolm
That character in the movie is my favorite.
He quit because OpenAi's culture didn't match his good version.
Interesting:
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
Too many rubbish talkings. It was written by AI?
Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.
OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.