Pantheon

The implications of AGI

(Mild spoilers for Pantheon follow)

Human beings are pretty remarkable creatures. At our best, we made it to the moon, figured out how to feed 8 billion people, and created Firefly. Our attempts to make AI more and more useful took off when the Transformer architecture was trained on massive amounts of actual humans talking and writing about things. Ever since, we’ve tried to make the models more and more like human brains, but we’re still at a point where it requires reading the entire internet and using the equivalent of 1.5 million human-years of calories to get to decent levels of performance, and it still requires considerable energy to generate new responses. Companies are already trying to apply lessons from the relative efficiency of the human brain in learning and thinking to AI. But what if we stopped tweaking algorithms to act more like humans, and just directly replicated a particular human brain’s structure, connections, and function? This is the premise of AMC’s animated series Pantheon, season 1 of which was just added to Netflix after years in US streaming purgatory. This show is great fun and a solid example of speculative fiction, so I wanted to take the opportunity to discuss it.

The center of Pantheon is the driving force that has moved much of human society from its beginning: the fear of death. By this I mean not only the end of our own conscious experience, but also the grief associated with losing those closest to us, and what it means to reach closure, to hold someone in memory, etc. This is already driving all kinds of moonshot attempts and overinflated valuations in Silicon Valley¹. Digital immortality by uploading a consciousness to the cloud is already something people are developing rudimentary versions of, as a way of providing loved ones with something in between being there and being gone.

But that’s just the first stop on this ride. In order to precisely map each of the neurons, they have to deconstruct and map the brain layer by layer with lasers, so the subject doesn’t survive. In a world where the best employees can live on forever as property of their employers, what kind of rights to a self do people and their families retain?² To what extent is the continued neural process of a person still that person?³ What would a digital person want to do, divorced from biological impulses?⁴ How does the alignment problem change when you’re dealing not with something you’ve developed from scratch but something with its own pre-baked motivations? If your body is just a digital avatar, is there any reason not to give yourself amazing hair?⁵

The show is a great example of how to do implications work in futures. Taking the initial change (uploaded intelligence), imagine the first-order possible consequences, then iterate the process until the world looks like a very different place. In this case, uploading would mean several things: the world’s most brilliant minds are able to dramatically improve their pace of work by simply adding more compute resources; companies and governments can obtain full rights over a person’s mind and output (and thus the dignity and respect for the rights of fleshy humans might decrease); different actors will have incentives to keep the existence of these entities secret vs disclose their existence; this would likely become the next front in the development of cyber warfare capabilities (potentially destabilizing the current balance of power), and so on. We call it the singularity because we can’t see what comes beyond it, but it’s still worth trying.

The show’s implications work is enhanced by the way it grounds its world in one we instantly recognize. Starting in essentially our present day, the series immerses the viewer in places, people, brands, etc, that we already know, or that serve as thinly veiled stand-ins. Like good back-casting, this connects a very strange future to the present by a series of plausible steps. Scenario timelines would do well to take notes.

The show as television — what can I say? It’s tightly written, fast-paced⁶, and has the best hacking scenes I’ve seen since 1995’s Hackers. If you are AI-adjacent and like nodding along to things you mostly understand and feeling smart because of it, it’s perfect for you. One of science fiction’s key contributions is extrapolating from changes in systems and structures so we can think about them more fully⁷, and this is a perfect example of that dynamic.

1. There’s a separate subplot I don’t want to spoil, but serves as (possibly intentionally) commentary on the way Silicon Valley expensively reinvents basic parts of life (but monetized), in this instance the family.

2. This theme of the “ultimate non-compete clause” is also a key part of Apple’s excellent Severance (I can only vouch for the first season at this point) and in discussions over a reanimated Peter Cushing in Rogue One.

3. Your stance on materialism likely influences your view of this.

4. For many people, this might be as simple as watching cat videos all day long without having to worry about eye strain.

5. Dear reader, there is not.

6. I couldn’t believe how much plot has occurred by the end of the second episode.

7. I cannot find the quote, but I’m convinced I once read something like “science fiction is about interesting rules, and fantasy is about interesting people”.