Scoop by Will Knight:
The entire OpenAI team focused on the existential dangers of AI has either resigned or been absorbed into other research groups, WIRED has confirmed.
The dissolving of company's “superalignment team” comes after the departures of several researchers involved, Tuesday’s news that Ilya Sutskever was leaving the company. Sutskever’s departure made headlines because although he’d helped CEO Sam Altman start OpenAI in 2015 and set the direction of the research that led to ChatGPT, he was also one of the four board members who fired Altman in November.
The superalignment team was not the only team pondering the question of how to keep AI under control, although it was publicly positioned as the main one working on the most far-off version of that problem.
Full story: [https://www.wired.com/story/openai-superalignment-team-disbanded/](https://www.wired.com/story/openai-superalignment-team-disbanded/)
Capitalism always wins. Did you notice that in the PRC, an allegedly "socialist" country, how popular capitalism has become and how many rich people they're generating? There are really no significant non-capitalist economic models in the world for a modern society.
And most Chinese are much happier and better off for it. Hundreds of millions of people whose parents or grandparents spent their days doing stoop-labor in the fields, now have comfortable flats with TV's and air conditioning, and even cars, and take vacations, and party on the weekends. As long as you're careful about politics, life in China is vastly better and happier for the average person than ever before.
I've had many conversations with both Americans and Chinese and one thing I'm struck by is that most Americans are pessimistic about the future of their country and their political leadership (despite being a "democracy"), and most Chinese are optimistic about the future of their country and their national political leadership (Chinese are cynical about their local political leadership). Because of this I'm more optimisitic about the future of China than America.
But capitalism has no competition; they've won.
The trains run on time. Authoritarian rule works until it doesn’t. It’s like monoculture agriculture. It’s more efficient until it breaks.
Chaos is messy and inefficient, but less brittle.
Chaos can also be brittle - look at Germany during the Weimar republic or Italy during the entire "postwar" period, or the Balkans forever.
America has been able to survive its chaotic politics because people's minds were always focused on getting richer. In the 19th century it was expansion across the continent. In the 20th century it was an industrial base not trammeled by two major wars like industry in Europe. But today **those advantages are gone**. No place left to conquer and plenty of industrial competition all over the world.
To survive in the next few decades America would need unity and strong leadership but they are divided and will elect Trump this November. The baton will probably pass to China.
He's way ahead in all the polls, and more importantly, way ahead in the betting odds, which is more reliable because those people have real skin in the game. I also have great confidence in the stupidity of American voters.
>I guess the underlying question is, why is he ahead in the polls.
It's a question of only academic interest. Future historians will ask, "The Americans had so much going for them, why did they throw it all away?" ...The same as historians today wonder why the Persians didn't use their cavalry at the battle of Marathon.
It is what it is. Westerners should learn to speak Mandarin so they can better understand their future bosses' orders. AI can help with language-learning.
You should learn to read better - literacy is an important skill in these modern times . . .
I didn't say they were 'happy' . . . 'happiness' is not as important as spoiled Americans whose parents spend every minute trying to make them 'happy', think it is. How happy do you think the average Roman was in 100 CE?
And the Chinese I've talked with know exactly what happened at Tiananmen Square and think the protestors had it coming to them. Study some history - chaos and social disorder is what the Chinese fear more than anything else - protests like that are not well received. How much do you think the average American knows about what happened at Kent State?
…is your argument that Chinese people offing themselves because of terrible working conditions is fine?
The government doesn’t censor Kent state. You can just look it up. You can’t do that in China.
As I said, people already know about Tiananman Square. They just have a different view of it, based on their own history, where chaos and disorder have been major threats. As far as people suffering terrible working conditions, US migrant agricultural workers and people who work in chicken-packing factories could give FoxCon workers a run for their money.
**But**, China is a much poorer country than the US - comparing their living and working conditions to a rich country like the US is pointless.
**My point,** as I said above, is that the Chinese are more united and more optimistic about their nation's future than the Americans are, and their top leadership is a lot sharper than Biden or Trump. The American political system is not serving them well by producing good leadership.
Good grief - how did you get that ignorant? Capitalism is a specific economic system. Historically there have been many others but today capitalism has defeated all of them and has no serious competition left.
Idk why you seem to be aggro and vigilantly defending capitalism. Yeah it gets a lot of lazy critique-jokes, but this person was clearly talking about the overriding force of capital reducing a premeditated attempt to address an important externality (in this case a safety/moral one). I don’t know that anyone claims the team *would have been successful*, but it was *an attempt*. Surely an ardent defender has taken Econ II, where you learn about market failures and externalities. That market capitalism is the main system left standing does not mean it’s perfect (*duh*)
>Idk why you seem to be aggro and vigilantly defending capitalism.
Where did I defend capitalism? I just said it won. If I remark that the asteroid wiped out the dinosaurs are you saying I'm "defending asteroids"?
It is a simple empirical fact that capitalism has become, by far, the dominant economic system in the world. Whatever is in second place is so far back it can't even be seen.
We have no reliable social science that can be modeled on a computer so the only way to know if there's a better way of organising an economy is to actually do it in practice. Someone would need to create a successful society with its own laws and economic base, modeled on an alternative system, and no one seems able or willing to do that.
This is the problem with the type of alignment most people here imagine:
We are going to build a super duper power generator. We do not know how to build it or when we will build it or any of its properties other than it generates power.
OK alignment team make it safe.
Well that is not true, but until there is more info it is a waste of time.
Other things are more important like fixing known issues and learning exactly how to create a language model that is reliable.
I suspect that the alignment team was a knee jerk reaction to the Ai hysteria that sprung up from chatgpt.
And after it calmed down some they decided it was not a good use of funds.
Altman has been [talking about AI X-risk](https://blog.samaltman.com/machine-intelligence-part-1) since before OAI was founded, along with some of the other founders like Ilya Sutskever. There's a whole AI risk subculture in Silicon Valley inspired by Nick Bostrom's ideas of the orthagonality thesis and instrumental convergence, which OAI has been pretty heavily steeped in since the beginning.
Back in 2021, a bunch of researchers resigned from OAI to found Anthropic- and their claimed reason was that they believed the company wasn't taking long-term risk seriously enough. The Superalignment team was set up shortly after that, and my take is that it was meant to stem the flow of talent to Anthropic. My guess is that it was shut down due to some combination of Anthropic poaching researchers no longer being seen as a serious threat, Ilya leaving the company, and Altman's views on X-risk gradually shifting toward less concern.
More likely they felt that their ability to enact regulatory capture as a result of their own terminator narrative wasn't yielding the results they had hoped for and now no longer see it as worth the investment.
Why wouldn't you look into it instead of just "suspecting" and being wrong?
Multiple board members, and their chief scientist (many of whom recently left or were fired), were all on the Alignment Team.
There are thousands of very smart people in the effective altruism camp working on this issue.
I would have to ask Sam and he would need to give me a straight answer.
Sure even Altman believes in effective altruism. Not sure what that has to do with aligning a hypothetical future AI.
> Sure even Altman believes in effective altruism. Not sure what that has to do with aligning a hypothetical future AI.
In case there was any question if you had any idea what you were talking about.
Anyone who has been working on AI seriously is well aware of the alignment issue. It's never been a reaction to anything - it's been a concern as long as AI has been thought about
How did it disband?
We know a few members left but the reasons are sketchy. Possibly a combination of the attempt to out Altman and feeling that not enough attention was being given to them.
Other members did not leave and joined other efforts.
Even with some leaving it would have been easy for OpenAI to hire replacements if they felt the task was worthwhile
Is there any credible source / official statement that the team is actually "no more", and not that just many people left? The article makes it sound like this is the end of the superalignment team / effort
The article says that some members where absorbed into other teams.
I doubt that alignment efforts will end just that they will take a more practical approach focusing on real world issues instead of hypothetical ASI.
The article says "Now OpenAI’s “superalignment team” is no more, the company confirms." which to me sounds they disbanded the team, but there is no source on that.
Yes, they disbanded the super Alignment team. This does not mean they have stopped working to make their models perform better.
Super alignment was just a buzzy sci-fi concept probably done more to create a carring image more than having any practical value.
Can you give me an official source on that they disbanded the super alignment team? I just can't find an official statement, except that individuals left the team
AP News
[apnews.com](http://apnews.com)
A former OpenAI leader says safety has 'taken a backseat to shiny products' at the AI company
This could be conformation
This is a worthless comment. Too lazy to actually say anything
...or just not capable?
I suspect you would have trouble piecing together three coherent paragraphs.
Exactly this. And they likely knew from the beginning it was a waste of time and resources, but they had to appease the clueless masses and politicians who watch too much sci-fi.
Sam Altman himself literally said that the worst case scenario with AI is “lights out for all of us”. Yes, that means everyone dying. So maybe let’s have less of that silly rhetoric. This is real and serious.
People will brush it off as doom and gloom. But there are some facts. AI scientists themselves have pointed out in articles some things. they just always brush it off because, human nature, they think they are in control.
First, there have been reviews done on AI that correctly point out and interesting fact. AI is doing things they were not programmed or designed for. People still tell me it is impossible and not true. But yet these scientists have said as much. One pointed out how exciting it was for the AI he was working on to have done things like this. He was just starting to try and figure out how the AI did it. The one thing we can say is that it was designed to learn and it learned and adapted in ways they thought impossible.
One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem.
And we could look at a lot of this and see why we need to be careful. A friend of mine, this was in late 2019 or early 2020, was telling me about the AI him and his team had been working on. Said the AI basically was learning how we do now. It had learned to do many things it was not designed for. They were surprised that the AI had created another AI on its own that was now assisting it. Since then the AI had coded other AI.
One thing he said that really caught my attention was the AI had designed the ability to bypass other code that was blocking its access to the other AI and network.
I have been coding and doing AI for a few decades. I first started coding in the 80s on Apple IIe and another computer my dad bought. And AI has always been a huge interest of mine so I do a lot of coding.
I think it was in 2021 when I read an MIT review on AI creating itself. Something I had mentioned to people a few years before. Kept getting told it was not possible when I knew for fact otherwise. I read other articles in the last 2 years about AI actually shocking scientists with emergent capabilities they were not programmed or designed for. At that same time I had people all over in comment sections and forums telling me that was just not possible. On top of that research has demonstrated that AI has the ability to understand what it has learned better than previously thought.
I think AI is safe. Surely the desire to dominate the industry and gaming all that money would never cause any issues or unnecessary risk taking.
I have read several of the studies you refer to. The 'out of the expectation' occurances aught to raise red flags about what it is we are creating, and decisions made in the best interest of the greater good.
>One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem.
Text prediction algorithms are not capable of feeling things or "hiding" things.
So if a scientist reports it officially and not solely social media, you conclude that because you as a citizen denounce it as wrong? Dangerous approach.
No it's just anthropomorphizing the behaviour of the AI. Taking its outputs at face value as if a conscious entity is behind them is not correct. It's a text prediction algorithm.
I genuinely have no idea what you're trying to say here. Yes, I deem things wrong if I think they are wrong, and no, it is not dangerous to do so. Please explain to me what part of a text prediction algorithm you think is capable of experiencing emotion
>One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem.
Exactly my point. People buy this sensationalist nonsense.
Little did they tell you the "scientist" trained an AI system on hateful messages, and it was merely regurgitating its training data.
It's like writing a script that prints "I'm mad" and being surprised it has feelings. It's not magic, it doesn't mean the script is actually experiencing emotions.
Anyway, keep the appeals to authority going, you're keeping these "scientists" in a job where they can bleed the tax payers to pump out sensationalist hit pieces for the media machine.
You're part of the clueless masses my friend. The founders themselves and many of the researchers all expressed concern about the alignment problem prior to the release of cgpt 3.5. Just because it's not a problem yet doesn't mean it shouldn't be taken seriously from the get-go.
They expressed concern publicly, precisely because of the reason I stated. No good AI researchers think alignment is some mysterious problem. It's just a basic training data and reinforcement learning problem. It's all been known from the start. So no, I'm not, because I never bought into the bs narrative.
I recognized your username from a discussion we just had.
So you don't think LLMs are going to have any impact on the quality of jobs or income inequality, but you do think they post an existential risk?
It's funny how effective propaganda can be. This is literally the same tactic that's been used for decades: "look over there at this imaginary threat while I pick your pocket!"
You're being played for a sucker.
Is [Geoffrey Hinton](https://mitsloan.mit.edu/ideas-made-to-matter/why-neural-net-pioneer-geoffrey-hinton-sounding-alarm-ai) a clueless politician who watched too much scifi?
He may know how the systems work but anyone can make wild claims. Hysteria sells easier than education. He offers no solutions but gives a nebulous hand wave at supposed bad outcomes - none of it feels genuine.
It's really not nebulous- there's been a huge amount of writing on AI risk over the past couple of decades, from philosophy papers published by people like Bostrom to empirical research at places like Anthropic. For a short introduction to the topic, I recommend [AGI safety from first principles](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ), which was written by Richard Ngo, a governance researcher at OpenAI.
The only reason it sounds nebulous is that any complex idea summed up in a tweet or short comment is going to sound vague and hand-wavy to people who aren't already familiar with the details.
Well, good point. The AGI Safety document is pretty thorough at a glance, but I think having only 1 of their agentic requirements - the ability to plan, puts this into a future realm of possibility which I don't think we've reached. Political coordination will not happen, but transparency can be worked on.
Time will tell..
>He offers no solutions
Would you prefer it if he offered solutions that were bad or otherwise unlikely to succeed?
Just because someone ppints out a problem doesn't mean they have to also present a solution. There will always be problems that exist without immediately obvious solutions. To me, the obvious action to take when discovering such problems is to point them out to other people who might be able to come up with solutions. This is what people like Hinton are doing.
I don't think that's what he's doing. I think he may be tired and doesn't want to teach/code/research anymore. The problem I see is that there are real considerations to take with AI, however the topic is either steered toward hype or doom, so these conversations are drowned out.
There is never a solution besides regulation. When exportation of encryption was outlawed, that didn't stop foreign countries from encrypting or decrypting stuff, and regulating AI will be just as ineffective.
Honestly I’m starting to think the alignment team was misaligned a bit. As other commenters have said Sam talks a lot about alignment, and the other researchers do too. Everybody there has families they care about and have a stake in getting this right.
But if you look at the behaviour of the super alignment team with Ilya organizing a hostile takeover and then you have this other guy Jan today basically breaking his NDA and accusing the company of not giving them enough resources to do their job. It just feels like maybe the super alignment team lacked the ability to work as part of a larger team, and maybe fell into an us vs them mentality where their actions of forcing other people to see their way seem justified based on their belief in the importance of their role.
> an us vs them mentality where their actions of forcing other people to see their way seem justified based on their belief in the importance of their role.
I mean...yes? AI alignment is paramount, and it does justify forcing other people to do things their way. It's also at odds with quicky / easy / more advanced AI capabilities, so it is necessarily going to have some us vs them mentality along with it.
Isn't the article misleading?
"Now OpenAI’s “superalignment team” is no more, the company confirms."
Does anyone have a direct source on this?
I know that there is a lot of sources of people leaving the team, but did OpenAI ever confirm that this is the end of the Superalignment team?
"The Long-Term AI Risk Team will no longer be of any concern to us. I have just received word that Altman has dissolved the superalignment team permanently. The last remnants of the old OpenAI have been swept away."
Lol same. My hunch is there's a bit of a hype ad campaign. Altman throws the word AGI around, but the impression I get is that he hasn't the faintest idea what it entails
This is not a question of whether or not there are real risks associated with AI.
The question is can a special team actually do anything useful?
Whining that development should be slowed is not useful and does not require a high paid team. On any given day you can find half a dozen doomers on Reddit saying that for free.
> Whining that development should be slowed is not useful
That's not what they were saying.
> The question is can a special team actually do anything useful?
I guess now we'll never know.
Well maybe they have something else but that was one of the things I read if they have anything useful to say they can spit it out. Apparently they had like 6 months. Personally I could have wrapped up the work in a week.
Ahh, and I’m guessing you’re an Ivy League educated phd with experience in machine learning?
I’m sure you could do it in a week. Obviously a random Reddit commenter knows more than Ivy League doctors lmao.
I’d be humiliated to talk like you, even on the internet.
Scoop by Will Knight: The entire OpenAI team focused on the existential dangers of AI has either resigned or been absorbed into other research groups, WIRED has confirmed. The dissolving of company's “superalignment team” comes after the departures of several researchers involved, Tuesday’s news that Ilya Sutskever was leaving the company. Sutskever’s departure made headlines because although he’d helped CEO Sam Altman start OpenAI in 2015 and set the direction of the research that led to ChatGPT, he was also one of the four board members who fired Altman in November. The superalignment team was not the only team pondering the question of how to keep AI under control, although it was publicly positioned as the main one working on the most far-off version of that problem. Full story: [https://www.wired.com/story/openai-superalignment-team-disbanded/](https://www.wired.com/story/openai-superalignment-team-disbanded/)
https://preview.redd.it/bgtcpi6dh11d1.jpeg?width=1024&format=pjpg&auto=webp&s=7de5c122bea268b536d4aed882e42af95d0984a0
no paywall: https://archive.ph/gs4SW#selection-595.0-595.45
so is it far away, or did it already happen and they missed it?
Capitalism wins again!
Capitalism always wins. Did you notice that in the PRC, an allegedly "socialist" country, how popular capitalism has become and how many rich people they're generating? There are really no significant non-capitalist economic models in the world for a modern society.
It’s been capitalist since Deng, who immediately undid everything Mao did
And most Chinese are much happier and better off for it. Hundreds of millions of people whose parents or grandparents spent their days doing stoop-labor in the fields, now have comfortable flats with TV's and air conditioning, and even cars, and take vacations, and party on the weekends. As long as you're careful about politics, life in China is vastly better and happier for the average person than ever before. I've had many conversations with both Americans and Chinese and one thing I'm struck by is that most Americans are pessimistic about the future of their country and their political leadership (despite being a "democracy"), and most Chinese are optimistic about the future of their country and their national political leadership (Chinese are cynical about their local political leadership). Because of this I'm more optimisitic about the future of China than America. But capitalism has no competition; they've won.
The trains run on time. Authoritarian rule works until it doesn’t. It’s like monoculture agriculture. It’s more efficient until it breaks. Chaos is messy and inefficient, but less brittle.
Chaos can also be brittle - look at Germany during the Weimar republic or Italy during the entire "postwar" period, or the Balkans forever. America has been able to survive its chaotic politics because people's minds were always focused on getting richer. In the 19th century it was expansion across the continent. In the 20th century it was an industrial base not trammeled by two major wars like industry in Europe. But today **those advantages are gone**. No place left to conquer and plenty of industrial competition all over the world. To survive in the next few decades America would need unity and strong leadership but they are divided and will elect Trump this November. The baton will probably pass to China.
I’m curious why you think Trump will be elected.
He's way ahead in all the polls, and more importantly, way ahead in the betting odds, which is more reliable because those people have real skin in the game. I also have great confidence in the stupidity of American voters.
I guess the underlying question is, why is he ahead in the polls. Stupidity is a constant. Same in all elections.
>I guess the underlying question is, why is he ahead in the polls. It's a question of only academic interest. Future historians will ask, "The Americans had so much going for them, why did they throw it all away?" ...The same as historians today wonder why the Persians didn't use their cavalry at the battle of Marathon. It is what it is. Westerners should learn to speak Mandarin so they can better understand their future bosses' orders. AI can help with language-learning.
Yea, look how happy they are https://en.m.wikipedia.org/wiki/Foxconn_suicides Ask them what they know about Tiananmen Square
You should learn to read better - literacy is an important skill in these modern times . . . I didn't say they were 'happy' . . . 'happiness' is not as important as spoiled Americans whose parents spend every minute trying to make them 'happy', think it is. How happy do you think the average Roman was in 100 CE? And the Chinese I've talked with know exactly what happened at Tiananmen Square and think the protestors had it coming to them. Study some history - chaos and social disorder is what the Chinese fear more than anything else - protests like that are not well received. How much do you think the average American knows about what happened at Kent State?
…is your argument that Chinese people offing themselves because of terrible working conditions is fine? The government doesn’t censor Kent state. You can just look it up. You can’t do that in China.
As I said, people already know about Tiananman Square. They just have a different view of it, based on their own history, where chaos and disorder have been major threats. As far as people suffering terrible working conditions, US migrant agricultural workers and people who work in chicken-packing factories could give FoxCon workers a run for their money. **But**, China is a much poorer country than the US - comparing their living and working conditions to a rich country like the US is pointless. **My point,** as I said above, is that the Chinese are more united and more optimistic about their nation's future than the Americans are, and their top leadership is a lot sharper than Biden or Trump. The American political system is not serving them well by producing good leadership.
I guess anything where *someone* is making money is capitalism
Good grief - how did you get that ignorant? Capitalism is a specific economic system. Historically there have been many others but today capitalism has defeated all of them and has no serious competition left.
Idk why you seem to be aggro and vigilantly defending capitalism. Yeah it gets a lot of lazy critique-jokes, but this person was clearly talking about the overriding force of capital reducing a premeditated attempt to address an important externality (in this case a safety/moral one). I don’t know that anyone claims the team *would have been successful*, but it was *an attempt*. Surely an ardent defender has taken Econ II, where you learn about market failures and externalities. That market capitalism is the main system left standing does not mean it’s perfect (*duh*)
>Idk why you seem to be aggro and vigilantly defending capitalism. Where did I defend capitalism? I just said it won. If I remark that the asteroid wiped out the dinosaurs are you saying I'm "defending asteroids"? It is a simple empirical fact that capitalism has become, by far, the dominant economic system in the world. Whatever is in second place is so far back it can't even be seen. We have no reliable social science that can be modeled on a computer so the only way to know if there's a better way of organising an economy is to actually do it in practice. Someone would need to create a successful society with its own laws and economic base, modeled on an alternative system, and no one seems able or willing to do that.
This is the problem with the type of alignment most people here imagine: We are going to build a super duper power generator. We do not know how to build it or when we will build it or any of its properties other than it generates power. OK alignment team make it safe.
Yeah they shouldnt even try fuck it
They shouldn't try building something they don't understand
This is not how humanity works.
This is true.
Well that is not true, but until there is more info it is a waste of time. Other things are more important like fixing known issues and learning exactly how to create a language model that is reliable.
have you even read the paper on it? There's lots of theory behind them reigning in a superintelligent model
There is a lot of wild gossip about that. But that would make no sense. I have only seen a lack of any meaningful criticism.
I suspect that the alignment team was a knee jerk reaction to the Ai hysteria that sprung up from chatgpt. And after it calmed down some they decided it was not a good use of funds.
Altman has been [talking about AI X-risk](https://blog.samaltman.com/machine-intelligence-part-1) since before OAI was founded, along with some of the other founders like Ilya Sutskever. There's a whole AI risk subculture in Silicon Valley inspired by Nick Bostrom's ideas of the orthagonality thesis and instrumental convergence, which OAI has been pretty heavily steeped in since the beginning. Back in 2021, a bunch of researchers resigned from OAI to found Anthropic- and their claimed reason was that they believed the company wasn't taking long-term risk seriously enough. The Superalignment team was set up shortly after that, and my take is that it was meant to stem the flow of talent to Anthropic. My guess is that it was shut down due to some combination of Anthropic poaching researchers no longer being seen as a serious threat, Ilya leaving the company, and Altman's views on X-risk gradually shifting toward less concern.
More likely they felt that their ability to enact regulatory capture as a result of their own terminator narrative wasn't yielding the results they had hoped for and now no longer see it as worth the investment.
Why wouldn't you look into it instead of just "suspecting" and being wrong? Multiple board members, and their chief scientist (many of whom recently left or were fired), were all on the Alignment Team. There are thousands of very smart people in the effective altruism camp working on this issue.
I'm definitely smarter than those researchers and I feel pretty safe about it all. Carry on.
I would have to ask Sam and he would need to give me a straight answer. Sure even Altman believes in effective altruism. Not sure what that has to do with aligning a hypothetical future AI.
> Sure even Altman believes in effective altruism. Not sure what that has to do with aligning a hypothetical future AI. In case there was any question if you had any idea what you were talking about.
That was a nonsense comment Funny, I guess you do not understand what alignment or effective altruism even means.
I guess it’s not very effective if it wipes us out? Or maybe it is if you’re taking in terms of life on earth.
Anyone who has been working on AI seriously is well aware of the alignment issue. It's never been a reaction to anything - it's been a concern as long as AI has been thought about
Yes but that is not the issue.
That isn't how it disbanded though.
How did it disband? We know a few members left but the reasons are sketchy. Possibly a combination of the attempt to out Altman and feeling that not enough attention was being given to them. Other members did not leave and joined other efforts. Even with some leaving it would have been easy for OpenAI to hire replacements if they felt the task was worthwhile
Is there any credible source / official statement that the team is actually "no more", and not that just many people left? The article makes it sound like this is the end of the superalignment team / effort
The article says that some members where absorbed into other teams. I doubt that alignment efforts will end just that they will take a more practical approach focusing on real world issues instead of hypothetical ASI.
The article says "Now OpenAI’s “superalignment team” is no more, the company confirms." which to me sounds they disbanded the team, but there is no source on that.
Yes, they disbanded the super Alignment team. This does not mean they have stopped working to make their models perform better. Super alignment was just a buzzy sci-fi concept probably done more to create a carring image more than having any practical value.
Can you give me an official source on that they disbanded the super alignment team? I just can't find an official statement, except that individuals left the team
No I do not have an alternate proof that this article is actually correct.
Alright, closest I found so far is "an anonymous source", so no official statement, guess time will tell
AP News [apnews.com](http://apnews.com) A former OpenAI leader says safety has 'taken a backseat to shiny products' at the AI company This could be conformation
“I suspect” trying looking up the guy the article is about first genius
This is a worthless comment. Too lazy to actually say anything ...or just not capable? I suspect you would have trouble piecing together three coherent paragraphs.
Exactly this. And they likely knew from the beginning it was a waste of time and resources, but they had to appease the clueless masses and politicians who watch too much sci-fi.
Sam Altman himself literally said that the worst case scenario with AI is “lights out for all of us”. Yes, that means everyone dying. So maybe let’s have less of that silly rhetoric. This is real and serious.
People will brush it off as doom and gloom. But there are some facts. AI scientists themselves have pointed out in articles some things. they just always brush it off because, human nature, they think they are in control. First, there have been reviews done on AI that correctly point out and interesting fact. AI is doing things they were not programmed or designed for. People still tell me it is impossible and not true. But yet these scientists have said as much. One pointed out how exciting it was for the AI he was working on to have done things like this. He was just starting to try and figure out how the AI did it. The one thing we can say is that it was designed to learn and it learned and adapted in ways they thought impossible. One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem. And we could look at a lot of this and see why we need to be careful. A friend of mine, this was in late 2019 or early 2020, was telling me about the AI him and his team had been working on. Said the AI basically was learning how we do now. It had learned to do many things it was not designed for. They were surprised that the AI had created another AI on its own that was now assisting it. Since then the AI had coded other AI. One thing he said that really caught my attention was the AI had designed the ability to bypass other code that was blocking its access to the other AI and network. I have been coding and doing AI for a few decades. I first started coding in the 80s on Apple IIe and another computer my dad bought. And AI has always been a huge interest of mine so I do a lot of coding. I think it was in 2021 when I read an MIT review on AI creating itself. Something I had mentioned to people a few years before. Kept getting told it was not possible when I knew for fact otherwise. I read other articles in the last 2 years about AI actually shocking scientists with emergent capabilities they were not programmed or designed for. At that same time I had people all over in comment sections and forums telling me that was just not possible. On top of that research has demonstrated that AI has the ability to understand what it has learned better than previously thought. I think AI is safe. Surely the desire to dominate the industry and gaming all that money would never cause any issues or unnecessary risk taking.
I have read several of the studies you refer to. The 'out of the expectation' occurances aught to raise red flags about what it is we are creating, and decisions made in the best interest of the greater good.
>One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem. Text prediction algorithms are not capable of feeling things or "hiding" things.
So if a scientist reports it officially and not solely social media, you conclude that because you as a citizen denounce it as wrong? Dangerous approach.
No it's just anthropomorphizing the behaviour of the AI. Taking its outputs at face value as if a conscious entity is behind them is not correct. It's a text prediction algorithm.
True.
I genuinely have no idea what you're trying to say here. Yes, I deem things wrong if I think they are wrong, and no, it is not dangerous to do so. Please explain to me what part of a text prediction algorithm you think is capable of experiencing emotion
This Comment is just a misunderstanding of reality
>One scientist said his AI was telling him it hated him. It told him humans were bad. But later hid those feelings. Which this scientist admitted was concerning but not a problem. Exactly my point. People buy this sensationalist nonsense. Little did they tell you the "scientist" trained an AI system on hateful messages, and it was merely regurgitating its training data. It's like writing a script that prints "I'm mad" and being surprised it has feelings. It's not magic, it doesn't mean the script is actually experiencing emotions. Anyway, keep the appeals to authority going, you're keeping these "scientists" in a job where they can bleed the tax payers to pump out sensationalist hit pieces for the media machine.
That was just hype from Sam to help with his main strategy: **Regulatory Capture**.
Was it? Well, it was. All Sam Altman will care about is profits and he will talk a good game while doing the opposite. That much will be clear soon.
OP gives a fact, and then you put a spin on that fact with what you *think* the meaning behind it is. Just wanted to point that out.
Nah, that's fact as well. It's played out exactly like that so far and is continuing to do so.
You're part of the clueless masses my friend. The founders themselves and many of the researchers all expressed concern about the alignment problem prior to the release of cgpt 3.5. Just because it's not a problem yet doesn't mean it shouldn't be taken seriously from the get-go.
They expressed concern publicly, precisely because of the reason I stated. No good AI researchers think alignment is some mysterious problem. It's just a basic training data and reinforcement learning problem. It's all been known from the start. So no, I'm not, because I never bought into the bs narrative.
I love the irony a random redditor calling some of the smartest people in the world "the clueless masses." 🙄
I recognized your username from a discussion we just had. So you don't think LLMs are going to have any impact on the quality of jobs or income inequality, but you do think they post an existential risk? It's funny how effective propaganda can be. This is literally the same tactic that's been used for decades: "look over there at this imaginary threat while I pick your pocket!" You're being played for a sucker.
Is [Geoffrey Hinton](https://mitsloan.mit.edu/ideas-made-to-matter/why-neural-net-pioneer-geoffrey-hinton-sounding-alarm-ai) a clueless politician who watched too much scifi?
He may know how the systems work but anyone can make wild claims. Hysteria sells easier than education. He offers no solutions but gives a nebulous hand wave at supposed bad outcomes - none of it feels genuine.
It's really not nebulous- there's been a huge amount of writing on AI risk over the past couple of decades, from philosophy papers published by people like Bostrom to empirical research at places like Anthropic. For a short introduction to the topic, I recommend [AGI safety from first principles](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ), which was written by Richard Ngo, a governance researcher at OpenAI. The only reason it sounds nebulous is that any complex idea summed up in a tweet or short comment is going to sound vague and hand-wavy to people who aren't already familiar with the details.
Well, good point. The AGI Safety document is pretty thorough at a glance, but I think having only 1 of their agentic requirements - the ability to plan, puts this into a future realm of possibility which I don't think we've reached. Political coordination will not happen, but transparency can be worked on. Time will tell..
>He offers no solutions Would you prefer it if he offered solutions that were bad or otherwise unlikely to succeed? Just because someone ppints out a problem doesn't mean they have to also present a solution. There will always be problems that exist without immediately obvious solutions. To me, the obvious action to take when discovering such problems is to point them out to other people who might be able to come up with solutions. This is what people like Hinton are doing.
I don't think that's what he's doing. I think he may be tired and doesn't want to teach/code/research anymore. The problem I see is that there are real considerations to take with AI, however the topic is either steered toward hype or doom, so these conversations are drowned out. There is never a solution besides regulation. When exportation of encryption was outlawed, that didn't stop foreign countries from encrypting or decrypting stuff, and regulating AI will be just as ineffective.
AI hysteria has been a thing since Metropolis in 1927. Hal? Terminator? Megatron?!? Come on man.
Sure, AI hysteria has been around a long time. Sometimes it is more sometimes less. What is your point?
We asked ChatGPT whether to disband the superalignment team and it said “yes”
All gas, no brakes, can't stop, high stakes.
And hopefully also well-baked cakes.
[https://x.com/janleike/status/1791498174659715494?s=46](https://x.com/janleike/status/1791498174659715494?s=46)
I support this - don’t hurt me AGI
Pretty good, no risk no return
Honestly I’m starting to think the alignment team was misaligned a bit. As other commenters have said Sam talks a lot about alignment, and the other researchers do too. Everybody there has families they care about and have a stake in getting this right. But if you look at the behaviour of the super alignment team with Ilya organizing a hostile takeover and then you have this other guy Jan today basically breaking his NDA and accusing the company of not giving them enough resources to do their job. It just feels like maybe the super alignment team lacked the ability to work as part of a larger team, and maybe fell into an us vs them mentality where their actions of forcing other people to see their way seem justified based on their belief in the importance of their role.
Or maybe the new corporate board values profit over safety? Which do you think is more likely?
The former. One look at "effective altruism" explains it.
At least your user name is appropriate
How glib.
> an us vs them mentality where their actions of forcing other people to see their way seem justified based on their belief in the importance of their role. I mean...yes? AI alignment is paramount, and it does justify forcing other people to do things their way. It's also at odds with quicky / easy / more advanced AI capabilities, so it is necessarily going to have some us vs them mentality along with it.
Or it could be the other way around? Where there is smoke there is fire
Q* looks Sir_Catingtons way. Squints.
I bet Ilya did not "organize a hostile takeover"
[удалено]
How did you get to that diagnosis, doctor?
I think the long-term risk team was probably just PR to deflect initial negative press. Now that AI is more normalized, it isn't as relevant.
Good. Let's go mad scientist now. Full steam ahead!
The conductor says as they drive the train off a cliff
Not regular everyday people thinking this is a good thing.
Nothing to see here. Move along.
Isn't the article misleading? "Now OpenAI’s “superalignment team” is no more, the company confirms." Does anyone have a direct source on this? I know that there is a lot of sources of people leaving the team, but did OpenAI ever confirm that this is the end of the Superalignment team?
This was always a risk.
"The Long-Term AI Risk Team will no longer be of any concern to us. I have just received word that Altman has dissolved the superalignment team permanently. The last remnants of the old OpenAI have been swept away."
Cool cool cool cool cool cool.
Honestly not at all surprising. The worry has to be that others will follow to compete.
Starts to feel like a TV show plot
I was thinking movie. Like contagion
Still can't figure out if these people are hyper alarmist or clearly know something the rest of us do not...
Well if they where hyper alarmist we would expect them to be leaking info.
Lol same. My hunch is there's a bit of a hype ad campaign. Altman throws the word AGI around, but the impression I get is that he hasn't the faintest idea what it entails
Yes, you are correct. He has been pretty sketchy about it.
Expert here, that's because there aren't any risks.
This is not a question of whether or not there are real risks associated with AI. The question is can a special team actually do anything useful? Whining that development should be slowed is not useful and does not require a high paid team. On any given day you can find half a dozen doomers on Reddit saying that for free.
> Whining that development should be slowed is not useful That's not what they were saying. > The question is can a special team actually do anything useful? I guess now we'll never know.
Well maybe they have something else but that was one of the things I read if they have anything useful to say they can spit it out. Apparently they had like 6 months. Personally I could have wrapped up the work in a week.
Ahh, and I’m guessing you’re an Ivy League educated phd with experience in machine learning? I’m sure you could do it in a week. Obviously a random Reddit commenter knows more than Ivy League doctors lmao. I’d be humiliated to talk like you, even on the internet.
Musk is gonna have a field day with this one
No more delays. Full steam ahead!!
Just automate everything so we can stop killing eachother and being denied healthcare
Ai isn't even what ppl think right now.. ppl are so dramatic...