How did you jump to that conclusion? The agent will be limited by the capabilities under its control. We have the technological ability to cripple world now and we don't have the technological means to prevent it. Give one AI control of the whole US arsenal and the objective of ending the world. Give another AI the capabilities of the rest of the world and the objective of protecting it. Would you feel safe?
> We have the technological ability to cripple world now and we don't have the technological means to prevent it
Humans have prevented it many times, but not specifically by technological ability. If Putin/Trump/Xi Ping wanted a global nuclear war, they'd better have the means to launch the nukes themselves in secret because the chain of command will challenge them.
If an out-of-control AI could discover a circuitous way to access nukes, an antagonist AI of equal capabilities should be able to figure it out too, and warn the humans in the loop.
I agree that AI development should be made responsibly, but not all people do, and it's impossible to put the cat back in the bag. The limiting factor these days is hardware, as a true AGI will likely need even more of it than our current LLMs.
Out-of-control AI is sci-fi fearmongering, it's not about worming through systems. It will be doing exactly what it was placed there to do. It will be a human failing that puts armageddon in it's hands. And since humans have NO MEANS to prevent armageddon (The predominant policy is in fact doubling down on destruction with MAD), there will be no way to place AI in command of this defense. The asymmetrical relationship between destruction and creation will mean there will never be a defense.
