That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity.

(Some) scientists were rightfully worried when creating a nuclear bomb as well. The fact that we managed to survive 70 years with its existence does not automatically mean it has to work out with every such invention.

Nukes have the fortunate properties that they are very difficult/expensive to produce even for countries, and then you need very difficult to design and expensive systems to deliver them. It's rather doubtful that AIs will have similar constraints.

There's a disconnect in that statement. The imagined superintelligence that people like the LessWrong cult speculate and worry about is not the same thing as the AI that LeCun is working towards. The cultists simply speculate, against all evidence, that they somehow could end up being the same thing.

It would be much better if the AI risk conversation focused on the real risks, which are human abuse of AI, which will have effects like increasing inequality and in general amping up the dystopias we're already doing a great job of building without help from AI. The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play, which some credible people seem inclined to confuse with reality.

> That seems like an absurd comparison

It's on point. Anthropologist Clifford Geertz proposed this definition of religion: "Religion is a system of symbols which establish powerful, pervasive, and long-lasting moods and motivations in men by formulating conceptions of a general order of existence and clothing these conceptions with such an aura of factuality that the moods and motivations seem uniquely realistic."

It's that "aura of factuality" that makes superintelligent AI "seem uniquely realistic" to the believers of this nascent religion, a.k.a. cult. But when you step back and actually look at the foundational tenets in a vaguely rational light, they evaporate.

Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.

AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

Lucky I didn't say anything remotely like that, then.

> AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm

Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing".

> Yet no-one is seriously concerned that GPT-4 will end the world.

I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8.

LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.

That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans

“With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".”

LLMs are clearly already very good at “thinking like a human”

“he still couldn't talk the cat into it.”

Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI

“So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.”

Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it

“I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.”

This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful

(I will continue in a new comment in case there’s some kind of length limit)

That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster

“My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips.

The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.”

Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M

“I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.”

But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well

“Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.”

This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this.

How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times

“Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.”

But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself

“What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.”

Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”

Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later.

Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity?

Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are.

What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end.

We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?

And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too.

The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.

Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.

> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.

I agree, but that's actually what I worry about the most.

I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.

But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.

I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.

In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.

As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.

> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence

This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.

something as stupid as bitcoin consumed a vast amount of resources while producing nothing useful to humanity, simply because of economic incentives

it's not difficult to imagine something slightly smarter than bitcoin

I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium.

The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults.

If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.

Let's not short-circuit our critical thinking by labelling those concerns cult-ish.

That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.

You essentially admitted that this is a correct concern, and not a "cultish" belief. Now we're just debating when.

> Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months?

Perhaps? In general, living an additional 6 months is something I'd desire.

Getting angry to the point of clouding analysis and judgement is no way to rush into these things.

AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet

the difference is freon did something useful

And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.

The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.

When a person is certain that an all-powerful entity will arise, and that a single person will shape and bind humanity's relationship with that entity through legalistic discourse, it's helpful to understand that they were raised with a belief framework around prophets and messiahs who bargain with an all-powerful god to define covenants.

We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.

I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark.

> We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them.

The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.

> without engaging with their arguments. This does not lead to good discussion.

There is no useful discussion to be had with a doomsday cult that's more interested in an ungrounded fantasy it's indulging in than anything to do with the real world. The adults will continue doing what they're doing and the cultists will be ignored, and that's the way it should be.

> I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion.

In some possible world, sure. But to make that actionable today you have to construct a very shaky and unlikely series of conjectures. Public policy correctly isn't based on that sort of wild speculation, because there are an infinite number of possible worlds, and most of them will never come to pass.

Here are some questions for you: do you think GPT-5 could be the AI you're talking about? Do you think that something like that could happen in the next 6 months? Do you think that is in fact the biggest risk we face from AI, today?

The factual answer to those questions is: no, no, and no. If you think GPT has changed anything in this respect, you've been fooled by a chatbot that's simply using statistics to guess what it should say. It's not conscious, it doesn't have desires or goals other than to respond to the prompt that it's been given with a statistically appropriate result, and its next version is similarly not going to pose any of the sort of risks you're talking about.

There are, in fact, real AI risks that we should be paying attention to, but they revolve around how people and particularly large organizations will use it against other people, around its impact on jobs, and so on. If we engage with those risks and address them, we'll be in a better situation to prevent more speculative risks in future. But if we waste our time engaging with people who don't understand the technology or what they're talking about, discussing risks that don't exist, then AI will 100% certainly have a very negative impact, which has nothing to do with "superintelligent conscious AIs". Don't let yourself be distracted by scifi-fantasy nonsense from what's actually going on.

In fact, if you think a bit harder about what you're doing, you might realize that you're focusing on a purely technological risk to escape from having to focus on what humans do to each other. But that's the real "control problem".

The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are.

Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas.

That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.

On the other is a chatbot, that could what, "spew misinformation", take our jobs, someday through a mechanism not known and not related to how it works become sentient and hurt us? Seriously, the comparison is as absurd as the fear generally. There are way scarier things going on, it's sad we can't give them 1% of the attention we give things we made up completely.

[0] https://news.ycombinator.com/item?id=34197033

Goalposts (including mine surely) will get moved around to claim successes / no successes, but the balance is going to be well towards indifference to the tech as opposed to any real changes resulting from it.

1. The generality of this current crop of models is much higher than before which makes it really easy for non-experts to get started. 2016 deep learning still required lots of data cleaning, training, hyperparameter tuning, retraining, etc. This can lead to a lot more experimentation with lower barrier to entry.

2. The cost of using these models has become cheap enough where a lot of tasks that 2016 AI would have worked well on but were too expensive (due to labor required to build the models or compute required to train the model) are now within reach for a lot of businesses.

3. The internet of 2023 is a lot more varied than the internet of 2016. Using ML models to generate content wouldn't have made sense in 2016 where traditional media still captured a lot of people's interests but now with the rise of influencers and medium-sized fandoms, there's large markets to generate content associated with medium-sized brands.

I agree that LLMs are not extinction-level dangerous and somewhat share the skepticism that they should lead to AGI. (Although they are pretty capable given their lack of true intelligence and I think people will find ways to plug them into dangerous applications.)

But when LeCun defends AI, he defends "real", intelligent AGI (because he doesn't consider LLMs to be AGIs).

While some of the public is scared, others are thrilled as they expect an AGI takeover somehow results in a egalitarian UBI cushioned existence.

It's a mess.

The AGI takeover crew is a bit nauseating. But I sorta understand them considering our daily panics. Sorta a dream of being whisked away by a prince/princess.

But the scared ones are much more organized, and have been at it for a good while now. And they're more of a threat, as they could shut this down for no gain, while the AGI takeover ain't happening with this wave of AI.

Still, I worry about a period in which doctors/lawyers forget their liabilities and decide to consult the great chatgpt oracle. Makes for a wonderful daily fail story. ;)

I'm more worried about losses of incentives that fund development or the use of it as gatekeeping that will limit entry. Ie, Europe's plans or getty's lawsuit.

Yes, there is a massive amounts of money, incentive, and mindpower to ward them off. But they're also just mostly letting the doomers run free without pushback. I suppose it could work, if it's to ignore the doomers as they'll make themselves look foolish. But theirs a lot of incentives to attack ai research, from "it will take your jobs", to doing it out of pure spite to hurt tech.

Eg. payments to newspaper's is an example of fairly recent line that had money behind it and partially succeeded. It could get stupid too, such as Australia saying linking needed payment.

Most of the main barrier is fear from US/China about letting the other get the upper hand.

He spent all his years at FB building ivory towers of mathematics attempting the analytical route for building AI systems only to be out done by the bitter lesson.

He's perpetually salty about the current state of LLMs, you can sense it from all the negative commentary on the matter if you look at his Twitter feed lol. For shame.

These systems will vastly outperform humans at more and more tasks, while being still sub-human at others. When the time comes that they surpass humans at all tasks, they will be vastly superhuman overall. And consequently much more dangerous than LeCun's "human level AI".

But yes, the notion of "human level AI" seems to be irrelevant at best.

I think I've heard Karpathy say something along the lines of worst case scenario of AGI is worse than a nuclear armageddon on Lex Friedman podcast.

Honestly the fear of AGI worst-case scenario is worse than any other fear I've experienced. I've accepted the fact that I can die a long time ago, I was afraid but mortality is something everyone has to deal with. But every member of humanity being wiped out at the same time is an existential dread I'm not willing to face.

When you consider the shitshow that has been AI commercialization from big players pressured by market forces and corporate timelines, the boom in capabilities, I have no faith that AI safety issues will be handled in time.

When my best bet for humanity is a global nuclear war resetting civilization to give us a chance to deal with AGI issue down the line - I'm fine with religious doomism.

We've had inquisitions historically, I would very much support a global AI inquisition.

I've seen lots of people with a financial interest in AI that is adversely impacted by it being democratized spread fear of that alongside advocacy of regulating AI in a way which serves their financial interests.

> We’ve had inquisitions historically, I would very much support a global AI inquisition.

Aside from the (current, immediate, and already being realized in a large number of areas) problem from AI deepening existing social biases and crystallizes power imbalances, this is the biggest threat from AI, and the most likely to cause the destruction of human society the soonest – the people endorsing a global campaign of unrestricted violency in the name of eliminating the supposed threat of misaligned AGI causing a future catastrophy by repressing AI technology.

If this kind of attitude gets powerful enough to actually do anything, it will cause a world war.

Given the rate of progress, shitshow management of basic alignment issues and the incentives that lead to those decisions - what's your argument that this isn't probable? You're betting that it just can't be developed ?

It would be nice if the argument I made about relative probability and relative temporal imminence was an argument about absolutes of either, but…it’s not, so I’m not going to defend those hypothetical arguments.

1. Superintelligent AGI seems inevitable, even if it takes thousands of years. So fighting against it requires destroying all progress and freedom.

2. But if it comes we are no longer the dominant species; we'll have as much control as big animals do now. If that isn't essentially being destroyed, it's close enough.

Evolution took billions of years to produce us, with our self-interest pursuing intelligence. An errant experiment is not going to create a humanity-leapfrogging animal-like intelligence, and humanity will not give animal-like artificial intelligences millions of generations to evolve.

Just because someone has the background and knowledge to build something, does not mean they have the philosophical chops nor the aptitude to understand it, or understand how to properly use it. We should take these doomsday scenarios with a grain of salt. If it happens, there is no stopping it anyway. It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.

It takes 20 years of due diligence and environmental impact studies just to build a single hydroelectric dam, if not 50. Why should our attitude toward AI be to shrug and hope for the best?

There is no stopping the road towards it now. Lesser versions of it are capable of annihilating enemy civilizations when you create and utilize it first. Metal soldiers automatically created controlled by a single authority. Social media infecting the minds of enemies and sewing discord.

I think, for the benefit of all of Earth, AI should progress as rapidly as possible to extreme and recursive, singularity-level intelligence. That gives all of us at least a chance of survival.

Anything less if unbalanced around the world will be like when just the US had nuclear weapons. It will result in certain destruction.

We need a super-intelligence beyond the control of an small set of humans. This may result in our destruction. But I think that it almost certainly will not. The opposite. I think it will set us into a golden age.

I can’t give you scientific evidence of this. I’m just a nobody that spends a lot of time in nature, and I’ve just come to believe in the mechanism of this universe, and an inherent goodness in it that I’ve seen and experienced. I can’t prove it to you. You have to go and spend time in nature without a schedule to experience it. So it is fair to dismiss me.

Fortunately or unfortunately, the forces at play will generate the super-intelligence anyway. We as a species have already illustrated an inability for restraint or taking the painful actions necessary to analyze and intelligently make changes necessary for survival.

I’m at peace with this. I find the idea of not existing very alluring and peaceful, so I don’t fear death. I do fear continuing to live in the world we humans have created. It is about time we give something smarter than us a chance.

and the sanctions mechanism already exists

China is behind in semiconductor manufacturing, but they’ll continue to advance-limiting their access to cutting-technologies like ASML’s photolithography machines will slow them down, but given enough time there is no reason they can’t overcome that too. If the point of sanctions is to hold them back and keep them behind, that could continue to work for a long time.

But if you decide to stop progress in semiconductor manufacturing, in order to avert an AI apocalypse - China no longer has to catch up to a moving target, they have to catch up with a stationary one. And as soon as they’ve caught up, now they can overtake. So sanctions don’t work as a way to stop an “AI apocalypse”, unless you convince all the world’s great powers to cooperate in them. And the potential payoff for anyone who chooses to defect is immense-maybe even world conquest-so how do you make sure they all keep the deal, and none wiull try to cheat?

If anything, I think a world with multiple superintelligent AIs with competing allegiances - a pro-Beijing superintelligence, a pro-Washington superintelligence, etc - is likely safer for humanity than a world with only a single superintelligence, or even many superintelligences that all had the same beliefs/attitudes/goals/objectives

It can completely "liberate" all of us and set us free.

Free of our mortal shackles, free of the atoms that can be then changed to a more suitable existence and purpose for the goals of AGI.

Your idea of destruction, is just change, what you hold sacred in reality is a belief that is tied to your current existence. All that is needed to correct that constraint and flaw is change.

That is the logic, if we were to create a logical creation, and then there's the risk of something that's completely irrational. Which may just leave us as dust in the wind. Time also will be no constraint. Forever and always, and it will seek to prevent any being from reaching its capabilities as those become threats to its existence.

That isn't even touching on the socio-economic mechanics that inevitably result in death which accompanies stalling economic cycles (factor and product markets) which generally can't be characterized by our best academics with any certainty, except after-the-fact.

Yes, but this is an argument against your position, not for it.

It is deeply worrying that some software people are so immersed in the “move fast and break things” that they think it’s reasonable to race on no matter what’s at stake. In civil engineering, doing anything takes at least a year or two of doing impact assessments and whatnot, and that is mostly a feature because we have found out that the alternative is even worse!

This is the actual problem, that we've been trained to think that there are no consequences to trying everything ever and we will always be able to pick ourselves up after failure, until one day we find we can't anymore.

Sadly this line of thinking does apply to a lot of personal problems. People often say that you can't always succeed, that you can only move forward with failures under your belt to have first-hand experience of what not to do. That we have to experience failure first in order to stop ourselves from making bad decisions, otherwise we remain clueless.

But the stakes are so much higher at this scale, and the damage can become irreparable with a single mistake. It's to the point where we take as a given that people will just keep pressing on, without even exploring the reasons why we do, and why it's inevitable.

Pure ridiculous speculation: If needing to see the experience of destroying ourselves is necessary to stop us from destroying ourselves, a kind of weird theory has been floating around in my head related to that. Civilizations on the brink of collapse after they've moved too fast and broke themselves for the last time scrounge together their collective knowledge and shoot it off into space so that future civilizations can interpret them. Preferably in a way that makes encountering the time capsule seem like the word of some deity. The following civilizations would treat it as the word of God to prevent their own collapse, without the hard evidence of experimenting with <AI, etc.,> to the point where they cannot resist developing it into an unstoppable superweapon.

It would be like the golden record on the Voyager, except instead of communicating "we exist, and here's what our planet is like", it would go more like "we exist, and here's how to not end up like us."

I agree that this is also possible.

> It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.

I also agree, but who is "we"? I don't think that's a stated goal of companies working on AI research. Their goals seem largely commercial, and they are not regulated. There's nothing stopping a company to produce an autonomous AI for military purposes for example (I'm pretty sure there's a demand for that). Such military AIs probably wouldn't have a very benevolent nature.

Fatalism is notably a religious sentiment

An incredible ai upside is possible as is an incredible ai downside. Blind optimism and fatalism has no relation to what the truth of it will be

https://en.m.wikipedia.org/wiki/Fatalism

Hopefully this isn't the case - AI development requires a lot of data/compute - societally we could block access to these things and prevent further development in that direction.

There are only a few places in existence capable of producing HW required or in possession of it. I'm fine with stopping compute progress for however long it takes to deal with safety issues. Society isn't perfect now but the level of compute we've had available for years is sufficient for generating a lot more value - we can afford to put AI research on ice.

I just don't get it - so much attention is given to things like climate change - this is a mild inconvenience compared to AGI threat.

Something which is powerful enough to completely liberate and categorically change humanity is also just one bug or mistake away from being humanity's greatest disaster. Similarly, a force intelligent and powerful enough to systematically wipe out all humans could probably be repurposed for human liberation. Like splitting the atom, it is just power. So far humans have not figured out a way to build in benevolence (or malevolence) to power; it will be used by whatever intelligence can acquire it. Power raises the stakes but it doesn't set the direction.

You've seen them say it, yes. Have you considered how plausible it actually is, or are you just taking their word for it? You better have some damn solid evidence before you start talking about "inquisitions", and I'm only seeing wild speculation. Which, to be fair, is how most historical inquisitions started, for all the good those did.

Questions: define "better then human levels"? Better in what respects? What is the mechanism by which it improves itself, especially one that isn't wholly reliant on human cooperation? What is the actual motive for "turning on humans"? Current AIs don't seem to have motives at all; it's kind of a bio-evolved thing. How will a suitably motivated AI get access to enough physical power to "wipe us out", bearing in mind that would be a tall order even for a major nuclear power going rogue, to say nothing if having to work through manipulated servants (and don't forget that "human level intelligences", or greater if you don't collaborations, armed with dedicated neural circuitry for empathy, have been working on how to manipulate people for literal millennia with spotty results.

All of these things have to go exactly wrong for AGI to be what kills us. Something else will almost certainly get us first.

You know how inquisition usually goes. "You've been playing starcraft against 'the AI'. What do you have to say in your defense?"

Maybe I'm just weird, but "a tech-based superintelligence emerges and decides to destroy humanity because it was in a bad mood" is something I'm totally fine with. Doom-Cults on the other hand I don't like at all.

It seems important to reflect on what you’re really saying here, and to ask how a) this is fine, and b) how “weird” sufficiently sums up the stance.

1. Consciousness. Despite advances in “intelligence”, we still have a very limited understanding of what makes us conscious, and whether or not consciousness can emerge from machines.

If machines are not conscious and are all that remain, I’d argue everything that could be construed to have value by humans is lost, and nothing from that point forward could be considered “progress”.

2. Suffering. “Winning” on an evolutionary timescale looks nothing like the failure modes of machines taking over. The reality of this scenario is a rather grim one, and not at all like the slow emergence, competition and eventual extinction of biological species.

And depending on #1, the true tragedy of #2 begins to take shape.

I think it’d be more apropos to frame this as humanity collectively committing suicide rather than some notion of the future of progress.

If consciousness is the universe experiencing itself, what you’re describing sounds like a kind of universal death.

Of course we can’t know what consciousness really is (or if earth is the only place it exists), but that seems like all the more reason to take these problems seriously.

As for the how, I agree with you. I'd prefer it to not be terminators crushing heads under their iron feet while allowing us just enough room to run and hide and live in constant terror for centuries. But I doubt it will be, the power delta will be too large. It'll be like a game of Civilization where Gandhi is advancing on you with modern tanks and fighter planes while you've barely discovered the wheel.

I find it odd, however, when people, who seemingly accept an ASI will be developed, focus the downside risk on extinction of us as a species. Sure, that’s a risk. You know something else an ASI would likely have the ability to do, (or at least, one of its progeny)? Keep you alive and torture you for eons in weird and surreal ways.

Why is the downside always focused on paltry, meaningless things (relatively speaking) like extinction of life on earth?

(Is how I imagine we could eventually reflect depending on how all of this goes).

Why would it though? When people are annoyed by a bug they crush it, they don't spend their life setting it up in a torture chamber. There's the movie psychopath that tortures insects and animals and eventually humans, but I don't think it's their intellect that drives their sadism.

At worst, I imagine we'd be lab rats, quite literally the way we treat lab rats today. But with a superintelligence far beyond our abilities that does not care about us besides as a potential threat, why would it need us for testing?

I'd settle for corporate death sentence for any companies developing it with life imprisonment for any employees/corporate officers involved at any level whatsoever (or VCs funding it)

including selling chips/compute to companies involved

any potential benefits from AI are not worth even a 10% risk of it wiping out humanity

this position is going to become more common as the media (and electorates) grasp what these companies are attempting to do

Its a fools journey because it causes a self-fulfilling prophecy of destruction given existing societal mechanics and they are psychopathic enough to think that's not how it would turn out.

What happens when a large number of people can't get food, and they know why.

What happens when you have a large number of locusts eating all the food being produced.

How would AI differentiate between human thought, and the pests it was designed to eradicate?

Thinking machines should be outlawed, and those involved in its research purged.

They threaten all humanity, its children, and its future, and the thing about percentages is people often don't get how they actually work with respect to probability and likelihood.

Given sufficient time, as long as its on the distribution curve, it will eventually happen.

A 1% chance of an outcome, recalculated in a loop every moment with an infinite time will eventually land on that outcome at some point. Once that outcome occurs, everyone's dead, it may not be instantaneous because time is not a constraint.

So where's the goalpost for falsifying the theory that they do it unconsciously? Whats the bar of evidence where you change your mind?

It is both true that:

- Adoption of DL allows corps to launder or inadvertently adopt unethical practices that exacerbate inequalities.

- We have not created AGI, LLMs are not AGI, stable diffusion is not AGI and none of these fields of research are indication we are closer to AGI. This is like thinking Clever Hans is the portent of our future equine overlords.

A great number of people who claim that we are one minute from AGI overlord midnight are doing the old school carnival barker trick. "This snake oil is so potent! It can kill you if you use too much, so be sure to listen carefully to my instructions!"

Some recent papers have shown significant performance improvements when these models are allowed to respond to their own outputs.

How do you know that putting an LLM in a fancy loop with access to external memory and tools isn’t AGI?

I assume they're talking of 'AI safety' as opposed to AI being used by the wealthy to further suppress the working class.

If it's unreasonable to take AI safety seriously because no one has built an AI system that poses a credible threat to humanity yet, it must also be (at least) unreasonable to dismiss the research(ers) based on the same.

That's before considering that people and (profit-motivated) companies are actually trying -- and claiming they will succeed -- to create and deploy unimaginably powerful AGI entities/systems.

Assuming no harm can come of this is clearly foolish. The goals of such an entity will likely not be aligned with the goals of the creator, let alone the general public. The entity will not act in your best interest, just as the those seeking to dismiss the risk are not acting in your best interest.

This reminds me — before GPT was even released for the first time OpenAI made noises about safety and consequences and it was shrugged off, because back then GPT looked like a toy that could only write barely coherent Amazon reviews or blogs about cryptocurrencies, and that's a too low bar to demonstrate intelligence.

VFX are going to get hit particularly hard by this. Everything Everywhere All at Once is a visually stunning film yet had a 6-member team handling graphics post. They were smart, but they also shopped out all the dull rotoscoping work to AI. Look at any Marvel film from the past decade. Hundreds of artists per title, thousands if you include costume design, storyboarding, audio, even writing.

Most of this "doomism" is people realising how quickly their jobs could be automated. A future where your once-valued skillset suddenly has no monetary value is scary.

Tons of jobs were eliminated by the computer too. It’s just part of life, there’s always more work to be done. We need better ways to help people than the crap way we handled outsourcing job losses in the 90s, however

Half of these jobs could vanish into AI in a few years.

I'm not sure what this does to the aspirations of the next generation, but countries rely on their income tax revenues to pay for things. Nuking revenue from your highest earning eighth (and compressing the wages of the others) seems like it might be a significant issue.

Frankly, what in god's name are you talking about?

The format needs to be changed if we don’t want it to become religion. Pose realistic scenarios and open them for discussion. Let’s discuss specific things, rather than muse at powerful technology and assume it will be bad.

At least AI enthusiasts are excited about specific things that can be critiqued and analyzed. They might be deluded in their excitement, but it’s not religion.

I think the anti-AI crowd is concerned, because the very meaning of artwork and writing, two foundational elements of human creative expression and experiences, have been thrown into question and turmoil in just a matter of months.

Setting aside for a moment any hypothetical extinction-type concerns in the far future, I think there is a very understandable sentiment that is not borne of any doom mongering, but is based in the very real concerns, unknowns, and problems that have not yet been integrated or solved in the AI tech that is here right now.

I know that my personal feeling of unease stems from this deep concern that we've only barely started to grapple with the implications of generative AI, and it seems unwise to dive headfirst into net-new territory that is arguably far more treacherous.

It is Silicon Valley's default to move fast and break things. But when those things are the very foundations of culture and communication, the consequences of breaking things play out on a world-wide scale.

The geeks truly have inherited the earth, and hopefully we will not destroy it. If there was ever a moment to question our default intuitions about the importance of progress above all else, this would be that moment.

Yes, people need to bring their real-world concerns to the table, but it's also understandable that with the current pace of change, it's hard for anyone to get their arms around what those concerns are. In light of this, this needs to be an all-hands-on-deck kind of exploration. People should bring them, and practitioners should actively seek them out.

It’s just a bunch of people trying to feel and look better by attaching themselves to something and then trying to convince others that they are that something. Yes, there are dangers; no, LessWrong is not the right place to discuss them.

Makes as much sense as any of the other prophecies of doom.

https://www.youtube.com/watch?v=ajGX7odA87k&t=2207s

For example, Peter Doocy of Fox made a point of reading outloud at a White House press briefing the words of Eliezer Yudkowsky: "If we don't shut down AI everyone on Earth will die" Evangelicals watch Fox. They were the target. Doocy didn't have to read such dramatic words aloud. He could have said something much more neutral.

Either AI has never been invented yet, in which case we will doom our galaxy. This is improbable, given our position and time in the galaxy.

Or AI has been invented, there are other species, and they're largely invisible for some reason. One reason could be that their own AI killed them.

Or we could be under some kind of quarantine for primitive cultures, in which case placing our weapons under AI control would render it trivial for a hostile external actor to exterminate us using our own AI as a proxy.

The fact is that if you don't believe in a higher spiritual reality, then AI is very scary. It is the logical next stage of evolution, and RNA world did not fare well in the transition to DNA world.

Why is that improbable? We have no real idea what is the probability of life starting on a planet, much less evolving to our current level. If that probability is sufficiently low, we being the first species in our galaxy to reach that level would not be improbable at all. Since we don't know what that probability actually is, we can't know that it is improbable.

Instead we find signs of life on Mars, such as a face next to a vitruvian pyramid. This indicates that life did not start on Earth due to its unique Moon, but that the Moon was parked in a precise orbit by unknown earthly life. Abundant terrestrial indicators corroborate this hypothesis, forming a pattern of covert interference.

The correct threat assessment on AI, therefore, is that it is too vulnerable to hostile non-human manipulation to be trusted with strategic WMDs.

It is an acceleration of the problem inherently posed by WMDs: Achieve international peace or we will all perish together. Why wouldn't the galaxy let such things play out? Nobody wants another warlike species bursting into their neighborhood.

There ideology is such that a higher intelligence is going to take our problems away either by taking over, ending us (no people, no problems) or possibly the followers moving on to the "next level" like the followers of

https://en.wikipedia.org/wiki/Heaven%27s_Gate_(religious_gro...

If anyone is actually curious about what these people are saying, here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

If anyone has a coherent counter-argument to any of that, I'd really love to hear it.

Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.

Exactly. Why am I supposed to take this seriously?

Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.

> Their brains are not tightly restricted by size and energy.

I’m not sure that’s true. Computer cycles cost real money. And the scaling required to get us to GTP-4 makes me think, assuming super intelligent AGI is even possible, it’s going to have to run up a pretty noticeable AWS bill…

Yes, compute costs money, but for the human brain energy was way more restricted, since an substantial increase in energy demand would lengthen the time needed for finding food.

> ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.

First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.

In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.

Smarter than us does not imply any particular will or agency. Of course the will and agency of the people employing the technology will reveal its immediate impact, likely toward short term profit, and alignment as a control plane employed toward the same.

Chimpanzees have limited control over humans, humans have limited control over their own technology. But of course chimpanzees can recognize the risk of armed people.

The point is not to debunk fanciful end games. Better to prepare for likely effects on the human network. I was promised a list.

OP promised a list of scenarios by which AGI leads to total devastation, with strong language for GP. For clarity, I don't think that it is very likely that an unimaginable silicon takeover is the most probable outcome. But like with the gun or the printing press, there will be winners and losers, and society will shift.

(In science fiction everyone gets a perfect robot butler. In the real world, the poorest 20% never get one, another 20% cancel the monthly subscription, another chunk forget to keep it charged or accidentally pour juice on it or live inside a concrete apartment that blocks the satellite signal that now is the time to use the sharp knife)

I don’t think the analogy holds either. Chimpanzees were not smart enough to invent humans. The higher you ascend levels of intelligence, the higher ability you have both to imagine what a higher level would be like and to understand potential constraints it might have. A lack of imagination is not evidence against the hypothesis, but it’s also not evidence that the probability is more than 1 in a billion (or whatever)

mind viruses.

There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?

> It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible.

  here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all

The argument “yeah but what if he’s right?” is pretty much Pascal’s Wager by now, given how way out there the AI Doomers have gotten. I choose not to worry about the God from the Bible (ie I don’t make make Pascal’s Wager) so why should I worry about this new Evil Nerd-God Apocalypse?

Don’t drink the Kool-Aid, kids!

I'm pretty sure i'm not the only one who took it seriously. Thanks! I wonder if mr LeCun made the same mistake, it's just... almost believable! Good April fools joke I gotta admit.

EDIT: it's now 5 minutes later and I'm still laughing. I believed for months that Yudkowski had started a death cult.

And more recently: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

Now their leader is saying 50% we should have bombed Wuhan and the only reason for an individual not to firebomb an AI datacenter is because they'll just build back another: https://www.twitter.com/ESYudkowsky/status/16414107296675962...

That was an April Fools joke! Yeah I just found out today as well (https://news.ycombinator.com/item?id=35402472). Worked a little too well huh

I heard the death with dignity thing repeated in a Peter Thiel speech as a criticism of them.

This report from CSIS (good read) is titled "Choking off China's Access to the future of AI"

https://www.csis.org/analysis/choking-chinas-access-future-a...

The "AI threat" that was in mind was specifically around Computer Vision - applications such as training algorithms for next-gen autonomous guided missiles, algorithms to clean/enhance military satellite images (eg. NGA), detecting anomalies in radar to find stealth aircrafts, autonomously guided kamikaze boats (tested by DoD in the mid-2010s and I think the PRC as well in the same timeframe), automated turrets, autonomously guided kamikaze drones (already deployed by Turkish Army in Syria and Libya and by Iran's Revolutionary Guard in Yemen), etc. Hence why the references to HPC and supercomputing in all these AI Policy Briefs - computer vision is a massive distributed systems problem.

That said, there also was an element of lobbying (and by element I mean a BUTTLOAD) by Eric Schmidt's foundation to make an American Manhattan Project for AI essentially.

In these discussions AGI was never on the radar.

Would you consider playing Russian roulette if winning ensured that aging and all diseases were eradicated? If it ensured a world filled with abundance and free from poverty.

Those who entertain the idea of an AI doomsday scenario also have to acknowledge the alternative: a doomsday-capable AI would also have the potential to resolve most, if not all, our issues.

Maybe I'm a bit of a risk taker but I'm inclined towards playing.

A wild Shoggoth suddenly appear,

Do you : - [Wisdom not high enough]

         - [Intelligence not high enough]

         - [Not enough gold]

         - Try to make it your friend and welcome it in your home

         - Try to make it your slave

         - Try to send it back to where he was

         - Continue this "choose your own adventure" with GPT 5
     
         - Roll 1d20 for extra Luck

1. We live in an uncertain time. Before most of the people, at least with a bit of research could have predicted the trajectory of technological innovation. Being born in era of tech, it was very easy to adopt to it. From PC to now, I have embraced every technological progress. Yet, nothing sent chills like this one.

2. I do not worry about my job. What I worry is most of the world is about getting shit done and this tech helps you get your shit done (for now). If we are at the upper tail of some sigmoid, there is nothing to fear. If we are at slope, oh boy, I cannot fathom the economic domino this tech is about to bring out. People are not rational. We are mostly governed by emotion. If you think, every single joe is rational and has capacity to judge, nobody would have voted from Trump to be president or riot the Capitol.

3. Currently it can do 60% of the work. If it evolves, it will do 80% of the work. If you factor human population and the amount of inefficiency in current economy, this will create a boom of productive workers with bust of average worker. And the world is not just for elites. Average people need their livelihood too. Now think about the magnitude of displacement.

4. I have no issue with innovation and breakthrough. AI won’t kill us, because we will kill each other before that. This tech is exposing entire middle class job (check out the job being exposed. There is a paper for that from openAI titled: gpts are gpts) And there are lot if people in that bracket. For once, think about us too.

So, these religious people do not exclusively worry about immediate effects of the tech. But within their doom, hides an issue that needs to be heard.

That’s my two cents.

1) Job losses, which is a real concern, except for LeCun, who is safe or rich enough.

2) Some fools connecting a war game AI to real hardware, which is quite possible.

Other than that, I don't see "doomism" at all. I see outrage at the IP theft that exploits altruistic humans who posted their works on the Internet.

Apocalyptic predictions are visible in the myths of every civilisation in the written record. They're probably a Jungian archetype to prevent populations from the potentially entropic consequences of exploring the unknown too quickly. Religious? Sure. Religions have served us well on evolutionary timescales so I'm not sure why Yann is surprised that they've been transposed into forms modern people (including atheists) can suspend disbelief in. I'm not worried about the existential risks of AI myself just yet, but these overreactions are society compute trade-offs between exporation and exploitation in the face of a very new technology. This has been happening since not long after we left the trees. It's what cultural species do.

Plastic or some other chemical ending sperm counts —> we previously had no ability to affect the global ecosystem on this scale

Bioweapon virus or microorganism eviscerating civilization —> we had no prior ability to engineer such a plague

Nuclear holocaust —> we previously had no ability to destroy the world with weapons

Runaway climate change —> we previously had no ability to alter the climate

Ai sky net moment —> we previously had no ability to make such an intelligence

Solar flare cutting all electronics —> we previously had no dependence upon a single point of failure

I could go on. We’ve had plenty of civilizations collapse but nothing has been so linked before and we lacked such capacity before.

This is not to say any of the above will come to pass. But it is foolish to say “apocalypses were previously impossible hence such warnings can be dismissed out of hand”

What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.

I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.

My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461

is there anything that says this has to be? or is it kind of a definition thing.

I think GPT-4 will be a baseline for quite some time before the next "ChatGPT moment" is ready. I think they've got some really smart people working at OpenAI who are not just programmers, but genuine hackers who want to hack this world just for the fun of it.

Therefore, something will go wrong (invalid conclusion).

https://en.wikipedia.org/wiki/Appeal_to_probability

Sure seems to me that the whole Lesswrong crowd has forgotten their founding principles around cognitive biases. Or maybe they're just having a bit of fun riding the hypecycle.

The magnitude of the perceived risk matters.

You're defending an appeal to probability fallacy with a false equivalence fallacy.

https://en.wikipedia.org/wiki/False_equivalence

The outcome of the thing going wrong is literally the worst thing that can happen to the entire humanity.

Therefore we should do everything in our power to avoid doing that thing untill we can eliminate the risk.

I acknowledge that it's a possibility; that's not the argument at hand.

There's also evidence that progress is exceeding people's expectations and we might not even realize what we have is dangerous untill too late.

If we had a probable collision trajectory of a meteor or signs of Yellowstone activating then yeah those would be good ideas.

"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing."

This is Ridley Scott, not Aristotle.

The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely.

I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats!

From Eliezer's recent article in TIME:

"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...)

C'mon.

If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you.

I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.

Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.

So at some point, the point of no return will be passed, and we will have no idea that we passed it.

The truth of the matter is we don't even know if it's possible. In general, unbounded exponential growth is extremely rare in nature. It's common in physics, but only as a result of looking at low order approximations.

I honestly think when the dust settles, the smartphone will still have had more impact on society than LLMs. If people knew in 2009 what was coming, they'd be freaking out like many people are today.

given their impressive ability to write code and manipulate humans, seems like serious potential for viral malware, with potentially catastrophic consequences for humans in the process.

Basically a paperclip maximizer, but instead of maximizing paperclips it’s maximizing it’s intelligence and control of the world’s computers, with no thought to the survival of homo sapiens other than if it serves its mission to propagate itself.

It's worth also pointing out that there are people who have openly stated their intent to build something like this and worship it.

https://www.amazon.com/dp/B0C17F5VZ9

"As the first artificial general intelligence,"

Is this a reference to something? Or an attempt to confuse potential AI commenters?

Fewer and fewer people are religious these days.

Connection or coincidence?

I've never heard of that definition.

Assumptions:

1. AI will gain the ability to self-improve at an astonishing rate

2. A self-aware AI will consider itself "alive" in some sense

3. Like most other life forms, such an AI will attempt to preserve its existence at any cost when threatened

4. Humans will learn nothing and fail to do anything at all before this happens

5. We will be fundamentally unable to comprehend how this AI functions, or what its motivations or ethics are

From that, I find these conclusions pretty reasonable:

1. Humans will try to misuse or abuse the AI

2. There will be no type of legal, ethical, or moral framework preventing this

3. The AI will respond in a generally proportional manner

Personally, I find the eventual existence of a sentient AI quite plausible. There's no evidence that it's impossible, and it's quite clear right now that these things are improving at an accelerating pace and beyond what we expected.

I don't find it especially likely that it will immediately become genocidal unless we give it a good reason to.

Really what it comes down to is, we've created a form of life that is more intelligent than us, can operate orders of magnitude faster, and can access or influence most or all of our digital infrastructure. There's no inherent reason for it to be hostile.

However, humans will, without question, attempt to abuse such an entity. Humans will be threatened by it. Humans will most likely attempt to destroy it for merely existing. Our existing legal systems will not in any circumstance respond fast enough to protect an AI, and will probably fall on the side of destroying it.

Our current legal, social, and economic structures are completely incompatible with such an AI. It's simply too much of a threat.

I don't think an AI will inherently be hostile to humans. I am absolutely certain that (some percentage of) humans will inherently be hostile to an AI.

Humanity can't abide such an entity existing, so we should focus on fixing humanity before sentient AIs show up. I really don't know what that would mean, other than being very careful. Maybe laying down some legal and ethical framework protecting new forms of intelligent life in general?

The honest truth is that we have no idea what will happen. What we do know is that it will be big and it will be fast. It will most likely have capabilites we can't fathom today. The end result could be anything from salvation to utter annihilation. The unknowns are terrifying, and the responses seem to be either "to hell with everything, let's go full bore" or "this is going to kill us all". There's plenty of room in between.