The AI apocalypse may not look how we expect
Ask the average person on the street what they think about when they hear stories of AI "killing us all", and I suspect you'll hear of AI drones flying around with lasers, or perhaps Terminator-style robocops.
Such an outcome is certainly possible, but there are other less obvious apocalypses that I think are more likely.
It isn't the norm to have modern healthcare, supermarkets full of food, or a functional global financial system. It's not a given that the planet is able to sustain 8 billion people (a 4x increase in the last 100 years). The fact that we're currently able to achieve these outcomes is an exceptional achievement and a historical anomaly.
Our modern life relies on underlying systems and social structures that have evolved gradually and carefully over time, and are complex beyond the possible understanding of any human. They are built on assumptions which AI threatens to take a wrecking ball to.
If the underlying systems that we rely on collapse, or are significantly degraded, the consequences to our lives could be catastrophic in ways that are hard to predict.
The internet is one example of a particularly vulnerable system. Being able to deploy software to the internet securely is something we've taken somewhat for granted, and is largely the result of a lack of resources on the part of attackers, rather than software security. In a world where modern LLMs are able to find complex vulnerabilities in an automated manner, operating internet-facing software securely may become significantly more expensive, or perhaps even impossible. Can modern human life be sustained in a world where internet services become much more expensive? What if the internet is only available in a highly degraded state? What happens if deploying complex computer systems in a secure manner becomes completely impossible?
Stepping away from technology, what happens to human social structures if intelligence and expertise become unnecessary? Can modern human life be sustained after those changes occur?