Over the weekend, a few friends and I had a small Zoom debate about AI, civilization, work, power and the usual cheerful subjects one discusses instead of simply opening a bottle of wine and talking about football.
We disagreed about plenty of things, but a few ideas survived surprisingly intact. I think these are worth writing down.
Intelligence May Become Cheap While Power Becomes More Concentrated
This is one of the strangest possible outcomes of the AI transition. Imagine intelligence becoming almost universally available.
A teenager in Nairobi, an accountant in Warsaw and a founder in São Paulo can all access something smarter than most specialists who ever lived. Wonderful. But an open-weight model does not magically give you: a nuclear power station, a gigawatt data center, 200,000 GPUs, a semiconductor supply chain, or the ability to borrow $100 billion.
If cognitive work increasingly scales with compute, then capital becomes an increasingly important input into intelligence. So we could end up with a genuinely bizarre world: everyone becomes much smarter, while the owners of the infrastructure required to produce intelligence become disproportionately more powerful.
AI democratizes capability and centralizes the machinery behind it at the same time. Those two things are perfectly compatible.
You Can Be Optimistic About 2040 and Terrified of 2029
I think a lot of AI arguments accidentally mix up two different questions: Where are we going? and How quickly are we getting there?
Suppose we eventually reach a world where machines perform most economically necessary work. That world could be fantastic. Cheap goods. Abundant energy. Personal medicine. Personal education. Robots doing dangerous and unpleasant jobs. Humans working less. Fine. Now imagine the transition takes thirty years. Institutions adapt.
New generations grow up inside the new economy. Education changes. Tax systems change. Companies reorganize. People have time to rebuild identities around something other than their job title.
Now imagine exactly the same destination, reached in three years. Entire categories of work lose economic value faster than people can retrain, governments can redesign social systems or society can decide what replaces employment as the primary mechanism for income, status and meaning.
Same endpoint. Completely different historical experience. This is why I don’t find it contradictory to be extremely optimistic about the long-term consequences of AI and extremely nervous about the next five to seven years.
The argument is often not about where we arrive. It is about speed.
Your Personal AI Should Know Who You Want to Become
Another thing we discussed that I keep thinking about: Most personalization systems optimize around who you already are.
Your preferences. Your habits. Your writing style. Your schedule. The things you usually buy. The projects you usually procrastinate on.
A sufficiently good AI can learn all of this and make you an extremely efficient version of your current self. I’m not sure that’s enough. The more interesting personal agent would also know: who are you trying to become?
Those are different datasets. Maybe you say family is important, but the agent notices you’ve worked every evening for three weeks. Maybe you say you want to become healthier, while repeatedly scheduling work over exercise. Maybe you claim you want to ship more products, while spending another week researching a decision that could have been tested on Monday.
A useful agent shouldn’t merely say:
Based on your previous behavior, here’s what you’ll probably do next.
Sometimes it should say:
You told me this mattered to you. Your behavior over the last month suggests otherwise.
That’s much more interesting. Of course, there is one annoying prerequisite.
Before the AI can align your behavior with your values, you need to decide what your values actually are.
Unfortunately there is no API for that yet.
Abundance Does Not Eliminate Status
I think this is massively underestimated in discussions about post-labor societies. The usual story goes something like: Robots produce everything. AI does all the intellectual work. Goods become extremely cheap. Everyone receives enough money. Problem solved. Economically, maybe. Psychologically, I don’t think so. Humans don’t only want food, shelter and leisure.
They want to matter. They want respect. Influence. Recognition. Competence. Someone to need them. And, inconveniently, many also want to be doing slightly better than the person next door.
You can give everyone identical houses, unlimited entertainment and robot-prepared sushi. Within approximately eleven minutes someone will invent a new hierarchy.
If intelligence stops being scarce, people who spent their lives deriving status from being smarter than others may have a particularly unpleasant adjustment.
What replaces it? Taste? Reputation? Physical achievement? Relationships? Originality? Ownership? Community leadership? Number of followers? Ability to run 100 kilometers for no sensible reason? Probably all of the above.
Material abundance solves material scarcity. It doesn’t automatically solve the much older human problem of meaning.
And Now We Get to the People Really Responsible for This
Science-fiction writers. Yes. I have developed a theory.
For decades these people sat around inventing extremely seductive nonsense. Brain implants. Artificial intelligence. Augmented reality. Mars colonies. Robot servants. Satellite weapons. Gestural interfaces. Connected consciousness. Synthetic humans. Immortality. Post-scarcity civilizations. Digital universes. And, depending on the author, either the glorious golden age of humanity or an unspeakable cybernetic hellscape.
Then millions of teenagers read this stuff. And some of those teenagers grew up. A few became engineers. A few founded companies. A very small number now control hundreds of billions or trillions of dollars. And what did we expect them to spend it on?
Accounting software? Of course not. They’re building the toys they wanted when they were fourteen.
Space rockets. Humanoid robots. Brain interfaces. Superintelligence. Mars. Virtual worlds. Gigantic orbital infrastructure.
The science-fiction story becomes a technical roadmap. The technical roadmap attracts engineers. The engineers build prototypes. The prototypes attract capital. Capital makes the ridiculous thing less ridiculous.
Eventually somebody stands on stage and says:
Today we’re announcing...
And suddenly a fantasy written in 1968 has a product page.
So I increasingly think science fiction is one of history’s strangest forms of self-fulfilling prophecy. Musk, Altman, Zuckerberg, Bezos? Victims of circumstances. They never had a chance.
Someone put the wrong books into their hands during adolescence. And I would particularly like to register a complaint against Stanisław Lem.
I suspect he is personally responsible for at least 20% of today’s GPU shortage.
So Where Does This Leave Us?
Probably in the same uncomfortable place where most serious AI conversations eventually end.
The technology can distribute extraordinary capability. It can also concentrate extraordinary power. The destination may be much better than the present. The transition may still hurt.
Material abundance may solve many problems without solving the human need for purpose. And the people building tomorrow are, to a surprising degree, trying to manufacture the futures somebody imagined for them decades ago.
I don’t know whether that’s comforting or horrifying. Probably both. But I increasingly think the useful response is neither panic nor passive optimism.
It is to understand what is changing, decide what kind of person you want to become inside that change, and retain enough agency that the future is something you participate in rather than simply something that happens to you.
Together is also considerably more fun. And slightly less scary.
P.S. About the Claude logo
And since we’re discussing literary conspiracies... Someone pointed out the resemblance between Claude’s starburst logo and Kurt Vonnegut’s famous drawing of an asshole in Breakfast of Champions.
Once you see it, unfortunately, you cannot unsee it. I have absolutely no evidence that Anthropic deliberately based its logo on Vonnegut’s anatomical contribution to American literature.
But if someday an internal brand deck leaks with the slide:
Visual inspiration: Kurt Vonnegut, 1973
I will not be entirely shocked. Perhaps science-fiction writers aren’t merely responsible for the technology.
Maybe they’ve been doing the branding too.