A FAIR perspective on generative AI risks and frameworks

Here is a summary comparison of the frameworks reviewed, highlighting their core strengths and focus areas.

The NIST AI Risk Management Framework offers a holistic AI risk management approach, focusing on governance, risk mapping, measurement, and management. The main strength of the framework is a broad, overarching perspective on risk management that integrates with organizational policies. NIST is also preparing A Plan for Global Engagement on AI Standards (NIST AI 100-5) and the Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Collectively, these frameworks provide a comprehensive approach to AI risk management, focusing on standardization, terminology, and practical guidance for secure AI deployment.

NIST’s Taxonomy and Terminology of Attacks and Mitigations outlines a framework for understanding adversarial attacks on AI, categorizing these threats and their mitigations within a structured taxonomy. It covers both predictive and generative AI, addressing the security and privacy challenges unique to machine learning. The aim is to standardize terminology and guide the secure deployment and management of AI systems, enhancing the understanding of adversarial machine learning risks and defenses.

The Databricks AI Security Framework proposes a holistic strategy for securing AI systems, identifying risks across AI system components and offering mitigation controls. The main strength of the framework is that it provides a comprehensive and technical focus across the AI system's lifecycle, promoting inter-team collaboration with actionable controls.

The FAIR-AIR approach focuses on quantifying generative AI risks through a detailed methodology, including AI risk posture management and the integration of FAIR for financial risk estimates. It covers five vectors of GenAI risks, emphasizing the importance of understanding and managing Shadow AI, foundational large language model (LLM) development, hosting, third-party management, and adversarial threats. The strength of the framework comes from its strategic, financial perspective on risk, aligning with broader organizational and cybersecurity risk management practices.

The AWS Generative AI Security Scoping Matrix is designed to help teams navigate GenAI deployment security with a flexible categorization of use cases. It distinguishes generative AI applications across five scopes with specific security disciplines, aiding in focused security effort prioritization, from a customer perspective. The main strength of the approach is its structured yet flexible framework, applicable across a wide range of GenAI applications and industries, focusing on tailored security considerations.

OWASP Top 10 for LLM outlines major security vulnerabilities specific to LLMs, offering practical prevention strategies. It addresses threats that are unique to LLMs like prompt injection and training data poisoning, emphasizing robust and new security practices. Its strength comes from its specificity to LLM security, providing actionable recommendations.

The Architectural Risk Analysis of Large Language Models from the Berryville Institute for Machine Learning delves into LLM-specific risks, advocating for security engineering from the design phase and suggesting regulatory measures. The research identifies 81 LLM-specific risks and emphasizes the top 10, focusing on attacker actions and design flaws. The strength of the research comes from its focus on transparency and detailed risk identification for LLMs and emphasis on better management and regulatory compliance.

Gartner's AI TRiSM framework focuses on trust, risk, and security management for AI implementations. It aims to ensure AI systems are reliable, safe, and compliant with ethical and regulatory standards. The framework advocates for a comprehensive approach across the AI lifecycle, addressing transparency, security, and ethical challenges. By prioritizing these pillars, TRiSM guides organizations in responsibly deploying and managing AI technologies.

The balance between strategic oversight and technical detail is delicate. While all the frameworks have their strengths, they also look at the problem from their own perspective or specific scope — each framework's strengths cater to different aspects of AI risk management and security.

Elastic InfoSec's risk management program is built around the FAIR quantitative risk analysis model. The FAIR philosophy is about conceiving risk as an uncertain event, the probability and consequences of which need to be measured. Risk is defined as the probability of a loss relative to an asset. Threats enter this definition as factors influencing the probability of a loss.

Elastic’s 2024 LLM Safety Assessment highlighted common LLM threat techniques, applications, and mitigations. Here is a summary of the threats we identified: 

• Prompt injection and jailbreaking are distinct attacks on AI models with different targets and implications. Prompt injection manipulates applications built on generative AI by combining untrusted user input with a developer's trusted prompt, exploiting the model to produce prohibited or unexpected responses. Jailbreaking, on the other hand, bypasses the model's built-in safety filters, aiming to unlock restricted capabilities and generate unsafe or unethical content. 

• Model poisoning occurs when adversaries intentionally tamper with the learning process of a model, often by injecting malicious data into its training set, compromising its performance or causing incorrect predictions or decisions. A specialized form of poisoning attacks is backdoor attacks, where a model is trained to produce incorrect outputs only when presented with inputs containing a specific trigger that would otherwise function normally.

• Training data poisoning involves introducing harmful or misleading information into the data set that trains a model. The goal is to corrupt the model's learning process, leading to biased, inaccurate, or malicious behavior.

• Model overloading attacks, including sponge attacks and input manipulation, degrade a model's performance or availability by overwhelming it with complex or recursive inputs. This strain on the system's resources can cause slowdowns or failures. 

• Model inversion attacks involve reconstructing sensitive information about a model’s training data. By carefully analyzing the model's outputs, an attacker can infer details about the input data or the data used to train the model.

• Membership inference attacks aim to determine if a specific data point was part of a model's training data set. Unlike model inversion attacks, which reconstruct information about the training data as a whole, membership inference specifically targets the presence of individual data points in the training set.

• Evasion attacks involve deliberately crafting inputs to mislead models into generating incorrect outputs. These inputs appear normal to humans but exploit the model's weaknesses to induce errors. 

Each of these exploits represents a different approach to interacting with or manipulating AI models — either the model directly or its training or testing data — at different stages throughout their lifecycle, potentially leading to various risks.

The threats posited above are just a few of the techniques that adversaries can use to achieve their goals. There are many different reasons why an adversary would attack an organization via an AI model, but a few could include: 

Sensitive data exfiltration: The potential consequence of initial exploits like prompt injection, where sensitive data is deliberately or inadvertently revealed or extracted through the model's interactions or outputs.

• Threat: Prompt injection, model inversion attacks, membership inference attacks

• Asset: Data

• Example loss event (quantifiable risk statement): The risk associated with an impact to the confidentiality of sensitive information as a result of unauthorized access and extraction of information due to prompt injection

Unintentional data leakage: The inadvertent exposure of confidential information, which can be a consequence of processed inputs or how the model was trained, such as incorporating too much identifiable information in training data.

• Threat: Issues more inherently tied to the model's development and operational processes, such as insufficient data anonymization, inadvertent model responses, model overfitting, feature leakage, and improper data segmentation

• Asset: Data

• Example loss event (quantifiable risk statement): The risk associated with an impact to data privacy as a result of accidental exposure of sensitive information due to oversight or failure in securely handling data inputs/outputs

Biased data/outputs: These can stem from initial exploits like model poisoning, but more often, they would arise from inherent biases in the data collection process itself.

• Threat: Training data poisoning, inherent data bias

• Asset: Data, model outputs

• Example Loss Event (quantifiable risk statement): The risk associated with an impact to fairness and decision accuracy as a result of the model generating biased or unfair outcomes due to skewed training data or malicious tampering

Harmful/unlawful outputs: Outputs that are inappropriate, harmful, or illegal, which could be directly elicited through exploits like prompt injection

• Threat: Prompt injection, adversarial attacks

• Asset: Model outputs, user trust

• Example loss event (quantifiable risk statement): The risk associated with an impact to user trust and compliance as a result of the model producing outputs that are harmful, offensive, unlawful, or violate policies due to manipulation or failure to adequately filter responses

Hallucinations: These occur when GenAI generates false, misleading, or fabricated information that does not align with factual accuracy or the input data provided. Hallucinations typically result from the inherent limitations or flaws in the model's design, training data, or operational logic rather than from external manipulation or attacks.

• Threat: Insufficient training data, complex queries

• Asset: Model outputs, user trust

• Example loss event (quantifiable risk statement): The risk associated with an impact to the quality of the product relying on the model outputs as a result of the model hallucinating and producing inaccurate, misleading, or false outputs

AI supply chain compromise: This refers to a broad umbrella of risk scenarios and vulnerabilities in the AI software supply chain, which can lead to unauthorized alterations, the introduction of malicious code, or data theft affecting AI models and their supporting environments. Much like traditional software supply chains, AI supply chains inherit many of its most common vulnerabilities. AI-specific threats include deliberate tampering with a model's training process (model poisoning) or the data it uses (data poisoning) — both of which can degrade performance or introduce biases, model theft, or model manipulation or abuse. The model manipulation or abuse of functionality, for example, occurs through misuse of the model's intended functionalities for malicious purposes. 

• Threat: Prompt injection, adversarial attacks

• Asset: Model outputs, system integrity

• Example loss event (quantifiable risk statement): The risk associated with an impact to system integrity and functionality as a result of attackers exploiting the model for malicious purposes, such as generating deceptive content or overloading the system

By defining risk scenarios, we can clarify the cause-and-effect relationship between the methods of compromise and their impacts. Potential threats represent the vectors through which the AI system's security can be challenged, while risk scenarios encompass the outcomes or losses that result from successful exploits. Next, we’ll take a look at a fictitious example based on one of the potential GenAI risk scenarios identified here. 

FAIR risk example scenario: FAIR loss chain for data exfiltration via prompt injection
The FAIR model breaks down risk into factors that can be quantified (in counts, percentages, or dollar figures) to estimate the probable frequency and probable magnitude of loss. From that we can generate a range of probable outcomes in financial terms to understand the loss exposure.

Truly unique to generative AI, prompt injection leverages the model's own processing to generate specific responses. This threat is novel because it directly exploits the non-deterministic nature and interactive, generative capabilities of AI models, which do not have a direct analog in traditional IT security. This represents a significant security concern as it introduces entirely new vectors for attack that traditional security measures may not address. 

• Model/data poisoning: While the manipulation of training data or model behavior is a concern in the context of GenAI, it parallels challenges in machine learning operations (MLOps) and data quality management. These are not entirely new concerns but rather adaptations of existing issues in data management and system integrity.

• DoS attacks: The concept of overloading a system to degrade performance or availability is well-understood in cybersecurity. For GenAI, the mechanisms might vary like complex queries, but the fundamental challenge remains consistent with traditional DoS mitigation strategies.

• Model inversion/membership inference: Data privacy and potential leakage concerns are not unique to GenAI and often arise from the ability to extract sensitive details from aggregated or anonymized data outputs. Whether through statistical analysis, pattern recognition, or other data inference techniques, the underlying information can sometimes be deduced, posing privacy concerns. This issue is akin to what is observed with model inversion and membership inference attacks in the context of GenAI, where the goal is to infer sensitive training data information from model outputs.

These risks primarily raise ethical, legal, and privacy concerns rather than traditional security threats. They relate to the consequences of the outputs generated by AI models rather than unauthorized access or system compromise. While these are not security risks in the conventional sense, they necessitate a comprehensive governance framework that addresses legal, ethical, and privacy considerations in generative AI deployment and use. Mitigation strategies require a multidisciplinary approach involving legal compliance, ethical guidelines, and robust content moderation and filtering mechanisms. 

The threat of supply chain compromise involving unauthorized alterations or the introduction of malicious code remains consistent with existing cybersecurity challenges that now extend into the AI domain. While the components at risk, such as training data sets and AI models, are specific to generative AI, the nature of the threat aligns with traditional supply chain security concerns. Traditional supply chain compromise strategies focus on securing the build chain and CI/CD pipelines that now incorporate AI components. This includes practices like securing code repositories, validating third-party components, and continuous monitoring for anomalies.

Managing the risks associated with GenAI is not just about addressing the immediate threats but also about understanding the broader context in which these technologies operate. As we’ve explored, the risks are multifaceted, ranging from novel challenges like prompt injection attacks to the evolution of familiar concerns, such as data privacy and supply chain vulnerabilities.

The most important questions we need to ask ourselves regarding AI security concerns, which are similar to those for any other product or service, revolve around three critical areas: 

1. Building AI products and systems: How are we building AI products? This involves scrutinizing the development processes, ensuring robust security measures are integrated from the outset and adhering to best practices in software engineering. 

2. Deploying and integrating AI into the existing landscape: How are we deploying AI products? Deployment strategies must include secure implementation, regular updates, and monitoring to safeguard against vulnerabilities.

3. Using AI-based third-party services: How are we using AI third-party services? It is essential to evaluate the security protocols of external vendors, ensuring they align with regulatory and compliance requirements as well as our own standards.

On Elastic’s InfoSec team, our approach to generative AI risk management is rooted in the FAIR quantitative risk analysis model, which provides a structured way to assess and mitigate these risks. By breaking down complex threats into quantifiable components, we can better understand the potential impacts on our organization and develop targeted mitigation strategies. The frameworks we reviewed offer valuable insights, but it’s clear that no single framework can address all aspects of AI risk. Instead, a comprehensive approach that combines strategic oversight with technical detail is essential.

As generative AI continues to integrate into our workflows and products, the key to managing its risks lies in continuous learning and adaptation. Our journey is ongoing, but you can see how we’re using generative AI for the better.