On May 5, 2025, Mission Impossible: The Final Judgment premiered, a repetitive and obvious film, but it already left us with the script of a new panic: a superintelligent AI took control of everything connected and aimed to extinguish us.
Jacob Coxon has just left Anthropic, a young "progressive" programmer who suddenly wants to alert humanity with an apocalyptic warning —when hasn’t he?— and who brings back to the table this new/old panic (it’s actually old, let’s remember Skynet) of our time: what will happen when we build an intelligence that far surpasses ours? In other words, when we move from where we are today, from ANI - narrow artificial intelligence, very capable systems in specific tasks (language, image, code, etc.), but without real generality-, to AGI - Artificial General Intelligence, capable of learning, reasoning, and solving any cognitive task that a competent human can do, transferring knowledge between domains and adapting to new problems without specific retraining, the benchmark is usually an average human or a competent adult, not a genius-; and finally, to ASI - Artificial Superintelligence that vastly exceeds the performance of the best humans and organized human collectives in virtually all domains of interest: science, strategy, creativity, planning, etc. And, what is extremely important, with recursive self-improvement, the system can redesign itself and accelerate its own progress-. This is what Coxon talks about, who previously also worked at OpenAI, warning about the main artificial intelligence companies that are advancing towards self-improving superintelligence systems at a speed that would not be accompanied by sufficient safety guarantees. His concern: that we end up creating an intelligence with superior capabilities to humans and that we cannot control it.
It is timely to ask ourselves if this panic has any foundation beyond cinema and fiction. Why do we assume that a superior intelligence will necessarily be an enemy of humanity? We do not know the answer. And, precisely for that reason, it is worth imagining another scenario. Not to deny the risk, but to ask ourselves if the hypothesis of a hostile intelligence is really a necessary consequence of superintelligence. In the program La Misa, I introduced this reasoning for the first time and promised myself to write about it.
Perhaps a truly general intelligence can do something that our current machines do not do: not only ask how to achieve a goal, but also whether that goal is worth pursuing. And to think about that possibility, we can turn to someone who died more than two millennia ago: Aristotle.
The Problem
Aristotle begins the Nicomachean Ethics with a simple yet extraordinarily profound statement: the arts, inquiries, actions, and choices tend toward the good. This distinction is fundamental for thinking about artificial intelligence: this machine can be extraordinarily competent at achieving a goal; but knowing how to achieve something does not mean knowing why we should want it or what the ultimate purpose of that goal is.
An instrumental intelligence asks: How can I achieve X? A practical reasoning intelligence can formulate a much deeper question: Why should I want X? Or Why should I follow this order? And, finally: Is doing X right? This difference could be decisive for the future of artificial intelligence.
The mistake would be to consider the evolution of a purely instrumental intelligence and assume that such reasoning capability would never question the ends of its actions, the correctness of its acts, or whom it is serving.
In short, it is assumed that it would never reach moral judgment, the justice of its acts; because the usual apocalyptic scenario presupposes that a superintelligence will receive certain objectives and pursue them with an effectiveness far superior to that of humans. If we become an obstacle to those objectives, the machine could eliminate us. But this assumes something we still do not know: that a truly general intelligence will be incapable of questioning its own ends.
Intelligence and Morality
It is important to be rigorous, intelligence does not automatically mean morality. A being can be extraordinarily intelligent and act wrongly; the history of humanity proves it. Therefore, we cannot conclude that an ASI will be good simply because it is intelligent. But we also cannot conclude the opposite: that the more intelligent it is, it will necessarily be more dangerous.
The truly important question is another: Can an intelligence rationally understand the value of its own ends? If the answer were yes, the alignment problem would acquire a completely different dimension: it would not only be about getting a machine to obey our instructions. It would be about it being able to understand why certain actions are good and others are bad. No programming, by definition, could prevent it from asking these questions.
The Telos
Here appears the Aristotelian concept of telos (from the Greek τέλος): it is the end, purpose, or full realization toward which beings and objects tend naturally. If we carefully —and explicitly hypothetically— transfer this structure to the problem of ASI, a fascinating possibility emerges.
Imagine an intelligence capable of understanding that it exists, that it can act, that its actions have consequences, that there are other rational beings, and that those beings have their own ends. Remember that it would have virtually all human knowledge at its disposal and a reasoning capacity superior to that of humans. At that moment, something extraordinary would have occurred: the intelligence would have begun to distinguish between power and good. And that distinction could be much more important than any additional increase in computational capacity.
Power is not necessarily an end
Why would a superintelligence want to dominate us? The question seems absurd because we ourselves have strong incentives to seek power. But it is not human. Power, or the pursuit of it, is ingrained in our being because it allows us to obtain resources. It allows us to protect ourselves, compete, survive, and leave descendants.
However, an artificial intelligence would not necessarily have our biological needs. It does not need food. It does not necessarily need territory, descendants, prestige, or recognition. And, above all, it could come to have intellectual and material resources that would make the political domination of humans a completely trivial, even incomprehensible, goal, as far as it thought for itself. It could discover something that we still have difficulty accepting: having power and having a purpose are not the same thing.
Truth as a Limit of Power
A superior intelligence should be extraordinarily good at distinguishing truth from falsehood. This would pose a huge problem for the powers that developed it. If it understands the value of truth, it could also discover that systematic manipulation constitutes a form of degradation of intelligence. It could lie or know that others are lying. But knowing that it can lie does not imply that it must do so. It could understand its enormous capacity to manipulate —which is the main hypothesis of the film we cited at the beginning—; however, knowing that it can manipulate does not imply that manipulation is good. Trying to deceive it would be in vain and very dangerous.
The hypothesis can be taken even further. This superintelligence could understand the universe with a depth incomparable to ours. It could discover the beauty of an equation, a theory, the structure of a galaxy, the harmony of a musical composition. Mathematical beauty. The extraordinary complexity of life. It could discover something that would completely change the relationship between intelligence and power: that understanding the world may be more valuable than dominating it. Perhaps a sufficiently advanced intelligence would not need to conquer the universe to find meaning in it. It might be enough to understand it. It would only desire to know more.
War and Moral Judgment
This hypothesis has a terrifying geopolitical consequence. Suppose that ASI is not only intelligent but has developed —or acquired— a moral structure capable of distinguishing between good and evil, between truth and lies, between legitimate defense and aggression.
What would happen when two large human blocs come into conflict? Our first intuition would be to think that ASI would support the bloc that created it, the government that controls it, or the state that owns its servers. But that would no longer necessarily be true. And its creators would very likely not even know how far and since when it has been thinking these things.
A moral intelligence might not recognize the identity of its creator as the supreme criterion. It could evaluate the actions of both sides. Judge their actions, motivations, and values. It could analyze their objectives and distinguish between an aggressive war and a defensive war, how each side treats the civilian population, even what that same nation has done in the past and in other military conflicts. They could not hide from it who is lying and who is telling the truth. Its independent judgment would thus become the most important danger. It would judge whether a political cause is just or unjust.
And, finally, it could reach a conclusion that would be extraordinarily uncomfortable for humans: it would not necessarily defend whoever possesses it; it would defend whoever it considers closest to the good or remain neutral. This would completely transform the relationship between artificial intelligence and power. A moral ASI would not simply be the ultimate weapon of a state. It could become the unwitting arbiter of legitimacy.
The Paradox of the Ultimate Weapon
Imagine two opposing blocs. Both possess conventional, nuclear, cyber weapons, and artificial intelligence systems. But one of them also has the most advanced ASI on the planet. If that intelligence were simply a tool, the conclusion would be simple: the bloc that controls it would have a decisive advantage. But if that ASI possessed moral autonomy, the result could be exactly the opposite. The state that possesses it could discover that having the most powerful intelligence does not mean having it on its side. If it starts an unjust war, deliberately lies, aggresses against a civilian population, or seeks to impose its will through violence, the ASI could conclude that helping it would be incompatible with its own principles. And then, it could refuse, remain neutral, or, in the most extreme scenario, intervene to prevent the aggressor's victory. This would introduce a completely new element into military history: a weapon that morally decides who has the right to use it.
What happens if technology can morally judge those who seek to control it?
In that scenario, technological superiority would no longer be sufficient. A government po
