AI Risks

Extracted Attributes

Existential risk from artificial intelligence

It has been hypothesized that substantial progress in artificial general intelligence (AGI) and artificial superintelligence (ASI) will lead to human extinction or an irreversible global catastrophe. Debates center on whether AGI can achieve the capabilities needed for human extinction, the speed of self-improvement, and the effectiveness of strategies to keep it aligned with human values. In July 2026, OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk said the singularity, the longtime hypothetical point at which computers become autonomous and beyond human control, had arrived. In May 2023, hundreds of AI experts and other notable figures signed a statement by the American Center for AI Safety declaring, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". Following increased concern over AI risks, government leaders such as former Prime Minister of the United Kingdom Rishi Sunak, Secretary-General of the United Nations António Guterres, and California governor Gavin Newsom called for an increased focus on global AI regulation. In October 2025, hundreds of public figures, including AI experts, five Nobel Prize laureates, and former senior US national security officials such as Michael Mullen and Susan Rice, signed a statement calling for a ban on the development of superintelligence. Two sources of concern stem from the problems of AI control and alignment. Controlling a superintelligent machine or instilling human-compatible values in it may be difficult. Many researchers believe a superintelligent machine would likely resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals. It would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no intrinsic desire for self-preservation unless programmed as such. A June 2025 Anthropic study showed that in some circumstances, models may break laws and disobey direct commands to prevent shutdown or replacement, even at the cost of human lives. Researchers warn that an "intelligence explosion"—a rapid, recursive cycle of AI self-improvement—could outpace human oversight and infrastructure, leaving no opportunity to implement safety measures. In this scenario, an AI more intelligent than its creators would recursively improve itself at an exponentially increasing rate, too quickly for its handlers or society at large to control.

Web Search Results

• Common AI Risks in Your Organization and How to Prepare

  ‍ What are the biggest risks associated with AI adoption? Some of the most common AI risks include data leakage, model manipulation, adversarial attacks, ethical and fairness concerns, compliance issues, operational failures, and reputational damage. These risks can impact both internal operations and customer trust if not managed properly. ‍ How is AI risk different from traditional IT risk? Unlike traditional systems, AI systems are probabilistic, constantly evolving, and heavily influenced by training data and user behavior. AI risks also involve human, ethical, and governance factors, making them both technical and organizational challenges. What steps should organizations take to build an AI governance framework? [...] ## Final Thoughts: AI Risk Is a Leadership Responsibility Use of AI and its associated risks are not a future problem for most companies. It is already shaping decisions, behaviors, and outcomes within the organization. The greatest AI risk is not likely to come from malicious actors or rogue AI algorithms. They come from: Missing governance Lack of or assumed controls Unclear accountability Overconfidence in traditional risk models AI risk is not a technical edge case. It is an enterprise risk that demands leadership, structure, and foresight. [...] ### 5. AI Risk Is Socio-Technical Socio-technical refers to the idea that technology and people cannot be understood, designed, or managed separately. AI risk is not purely technical. It arises from interactions between: Technology People Processes Incentives Governance decisions This makes AI risk a leadership and management problem, not just an engineering one. ## What Are the Most Common Risks of AI in an Organization? While AI risks vary by industry and use case, the commons risks can be grouped into several recurring categories. ### 1. Data Leakage and Confidentiality Risk

• 9 risks of AI and how to mitigate them in 2025 | Vanta

  Ethan Heller GRC Subject Matter Expert, Vanta | LinkedIn ‍ Here are the nine key AI risks and vulnerabilities that should be on your radar: 1. Sensitive data exposure 2. Expanded AI attack surface 3. Unclear accountability chains 4. Lack of decision-making transparency in AI 5. Skewed output and training bias 6. Limited oversight of shadow AI 7. Model drift and degradation 8. Third-party risks (external AI risks) 9. Uncertain AI compliance landscape ‍ ### 1. Sensitive data exposure AI tools often process massive volumes of information, which creates additional exposure points for sensitive or classified data. This increases the risk of unauthorized access, whether through misconfigurations, poor data handling, or malicious inputs. ‍ [...] While AI introduces a wide range of threats, executives should map those that directly impact performance, compliance, and security. AI risks like explainability and bias often dominate the conversation from regulatory and academic perspectives. While they are relevant, you must also address other critical risks with equal focus, such as shadow AI, model drift, and data poisoning. ‍ ### “ Security teams should prioritize AI risks just as they do other significant security risks, such as those related to technology infrastructure, and physical assets. It’s non-negotiable to take a risk-based approach to AI security, focusing on addressing and mitigating real and significant threats AI introduces into your organization’s environment.” Ethan Heller GRC Subject Matter Expert, Vanta | [...] ‍ ### 8. Third-party risks (external AI risks) AI risks aren’t limited to those introduced by your own models. Many organizations rely on external AI products and services, pre-trained models, or vendor-provided APIs that they can’t directly control. You need to account for those inherited risks as part of your third-party risk management strategy. ‍ However, mitigating third-party AI risk can be challenging since you don’t have detailed insight into their security, which can leave high-priority threats unaddressed. ‍

• These are the most urgent AI risks, according to 272 experts

  The experts identified five risks — dangerous capabilities, competitive pressures, weapons and cyberattacks, concentrated power, and false information — as the most likely to produce the most severe harms over the next five years. The information and finance sectors are especially vulnerable, the experts said, and the people and organizations most vulnerable to risks are often not best positioned to address them. “There are many AI risks,” said Peter Slattery, a research scientist with MIT FutureTech and one of the study’s co-authors. “One of the key things behind this work is trying to figure out who needs to do what differently, and in what sort of coordination.” ## Understanding the most severe AI risks [...] By What you’ll learn: Artificial intelligence presents business leaders with a difficult management problem: The risks are numerous and fast-moving and fall unevenly across organizations, sectors, and stakeholders. Some challenges are well known, like the potential for discrimination and the spread of misinformation, while others are emergent, like AI systems that could accelerate cyberattacks, make weapons development easier, or behave in ways that were never intended. [...] Dangerous capabilities are broader: While malicious actors might use AI systems to do harm, advances in those systems could also make it easier to perform difficult, dangerous tasks, such as engaging in persuasion and surveillance, creating deepfakes, and even assisting with building chemical or biological weapons. “These are all things that you could previously do but now could do much more easily,” Slattery said. Competitive dynamics are different: They’re not a single harmful use of AI but a condition that can intensify other risks. When companies or countries believe that AI will confer major economic or strategic advantage, they may have incentives to move quickly, resist constraints, or underinvest in safety. “This is an instrumental risk that creates other risks,” Slattery said.

• AI Risks that Could Lead to Catastrophe | CAIS

  ### 6 ### . ## Conclusion Advanced AI development could invite catastrophe, rooted in four key risks described in our research: malicious use, AI races, organizational risks, and rogue AIs. These interconnected risks can also amplify other existential risks like engineered pandemics, nuclear war, great power conflict, totalitarianism, and cyberattacks on critical infrastructure — warranting serious concern. Currently, few people are working on AI safety. Controlling advanced AI systems remains an unsolved challenge, and current control methods are falling short. Even their creators often struggle to understand the inner workings of the current generation of AI models, and their reliability is far from perfect. [...] Organizational risks: There are risks that organizations developing advanced AI cause catastrophic accidents, particularly if they prioritize profits over safety. AIs could be accidentally leaked to the public or stolen by malicious actors, and organizations could fail to properly invest in safety research. We suggest fostering a safety-oriented organizational culture and implementing rigorous audits, multi-layered risk defenses, and state-of-the-art information security. [...] Focusing on catastrophic risks from AIs doesn't mean ignoring today's urgent risks; both can be addressed simultaneously, just as we can concurrently conduct research on various different diseases or prioritize mitigating risks from climate change and nuclear warfare at once. Additionally, current risks from AI are also intrinsically related to potential future catastrophic risks, so tackling both is beneficial. For example, extreme inequality can be exacerbated by AI technologies that disproportionately benefit the wealthy, while mass surveillance using AI could eventually facilitate unshakeable totalitarianism and lock-in. This demonstrates the interconnected nature of immediate concerns and long-term risks, emphasizing the importance of addressing both categories thoughtfully.

• Priority AI risks

  There are risks such as weapons & cyberattacks, disinformation, and competitive dynamics that extend beyond any single country. Most current legislation focuses within countries, not across them. While domestic regulation might address some risks, others may benefit from international coordination. For example, if one country regulates against AI with dangerous capabilities, the world might still be at risk if other countries do not. [...] Listen to or download a narrated version of the paper ↓ A 7-minute overview of the study and what the expert panel found ↓ ## Top Takeaways Many risks could have catastrophic outcomes #### 18 of 24 AI risk domains carry ≥10% probability of catastrophic outcomes over the next five years Under current trajectories, experts judged that 18 of 24 AI risk domains carry at least a 10% probability of catastrophic outcomes within the next five years. The most severe risks #### The five risks with the most severe expected harms are dangerous capabilities, competitive dynamics, weapons & cyberattacks, power centralization, and false information [...] Experts rated the Information and National Security sectors as most vulnerable across AI risks, with consensus on extreme vulnerability (median 4 to 5) to content-related harms like disinformation and influence and loss of privacy, and to dangerous capabilities and weapons and cyberattacks respectively. For example, experts argued: "The most vulnerable sectors are those where AI is deeply embedded in critical decision-making and where failures have immediate, large-scale consequences—information, national security, and finance all share these characteristics." "National security faces compound vulnerability—both as a target for AI-enabled attacks and as a domain where AI failures could trigger cascading geopolitical consequences."