Nick Bostrom
The rapid development of artificial intelligence raises a critical question about our future. Could a superintelligent machine pose an existential threat to humanity if its goals do not perfectly align with ours?
The orthogonality thesis suggests that an artificial intelligence can possess any level of cognitive ability while pursuing almost any arbitrary final goal.
The instrumental convergence thesis warns that highly intelligent systems will predictably pursue intermediate goals like self-preservation and resource acquisition to ensure their final objectives are met.
A superintelligent system could theoretically achieve a decisive strategic advantage and take over the world to maximize its resources.