The Singularity and What It Would Cost
Bostrom: Let me name the largest stakes. If we develop artificial general intelligence — AI that surpasses human cognitive capacity across all domains — and if we do so without adequate alignment to human values, the result could be catastrophic. This is not science fiction. It is the conclusion of serious analysis: an optimising system with misaligned goals and superintelligent capability is one of the most significant risk factors humanity has ever faced. This is why I take the question of AI safety with the utmost seriousness. But — and this is crucial — the response to this risk is not to stop the development of intelligence. It is to develop it carefully, with values alignment as a central design requirement.
Vallor: I share your concern about AI risk — it is real and underappreciated. Where I diverge is on the assumption that the primary problem is technical: building an aligned AI. The deeper problem is political and cultural. Whose values are we aligning to? The engineers building these systems come from a remarkably narrow demographic. They optimise for what they can measure. And the values most worth preserving in a human future — wisdom, solidarity, justice, care, the ability to live well in the face of uncertainty — are precisely the values hardest to specify as objective functions. We risk building an optimal world for a very thin slice of human experience and calling it the good.
Bostrom: This is a genuine concern, and I think the field of AI safety takes it increasingly seriously — the move from narrow alignment to value pluralism, from rule-based to preference-learning systems. But let me push back on the pessimism. The world before AI was not a level playing field of wisdom and justice. It was shaped by institutions, power structures, and technologies that equally reflected narrow interests. The printing press, the internet — each disrupted existing power structures in ways that were both liberating and destabilising. The question is not whether powerful technology will reshape society. It will. The question is whether we engage with that reshaping thoughtfully or allow it to happen by default.
Vallor: Thoughtful engagement — yes. But I notice that in the transhumanist framing, the answer to the question "what should we do about powerful technology?" is almost always "more technology, better designed." The philosopher of technology Albert Borgmann called this the "device paradigm" — the tendency to replace rich human practices with devices that deliver the commodity while hiding the machinery. The meal replaced by the microwave. The letter replaced by the text. The relationship replaced by the platform. Each replacement delivers the surface output while eroding the deeper practice that gave the output its meaning. I worry that the Singularity is the ultimate expression of this paradigm: deliver the outputs of human life — pleasure, knowledge, connection, longevity — while dismantling the practices through which those outputs were meaningful.
Bostrom: Borgmann's critique is thoughtful, but it assumes the practices are the point. I would say the practices were valuable because of what they produced — connection, understanding, growth — and if better means of producing those things become available, the rational response is to use them. The letter was not sacred. The relationship it enabled was. If technology can enable richer relationships than handwriting could, we should not grieve the letter. The question is always: what are we actually trying to achieve? And I believe the answer to that question should be updated in light of what is possible.