S-risk pathways in rough order of how likely I think they are

S-risk pathways in rough order of how likely I think they are:

- Partially aligned AIs. Imagine an AI that we've made to value living humans. Which, hopefullly, we will do! 

Now imagine the AI isn't entirely aligned. Like, it wants living humans but it's also been given the value by Facebook to click on Facebook ads. 

It could then end up "farming" humans for clicking on Facebook ads. Think the Earth being covered by factory farmed humans for Facebook ads. 

Except that it's a superintelligence. It can't be stopped and it's also figured out how to extend the life span of humans indefinitely, so we humans never die. 

Could happen for any arbitrary value set. 

- Torturing non-humans. Or, rather, not torture. Torture is deliberately causing the maximum harm. 

I'm more worried about causing massive amounts of harm, even if it's not deliberate and it's not the maximum. Like factory farming isn't torture, but it is hellish and is a current s-risk. 

So I care about more than just humans. I care about all beings capable of suffering and capable of happiness, in the broadest possible definition.

It could be that the superintelligent AI creates a ton of sentient beings and is indifferent to their suffering. 

I think this would mostly be it creating a lot of programs that are suffering but it doesn't care about. Think Black Mirror episodes. 

Generally, if something is indifferent to your suffering, it's not good. It's usually better if it kills you, but if you're useful to it, it can be really bad for you. 

- Malevolent actors. Think of what dictators currently do to dissidents and groups they don't like. Imagine they had control over superintelligent AIs. Or imagine they gave certain values to a superintelligent AI. 

Imagine what could happen if somebody asked a superintelligent AI to figure out a way to cause the maximum suffering to their enemies? Imagine if that AI got out of control. 

Or heck, it could also just be idiots. Within about a week of people putting together AgentGPT some kid in a basement gave it the goal of taking over the world. 

This is especially a risk with open source AIs. 

The population of idiots and sociopaths is just too damn high to put something so powerful out there for just anybody to use. 

- Accidentally flipping the sign. If we teach it our values, it's really easy to just "flip the sign" and optimize for the opposite of those. 

That's already happened, where an AI that was programmed to generate new medicines was accidentally switched and then ended up generating a whole bunch of poisons.