Welcome to OffSec's Advanced AI Red Teaming course. In this introductory module, we'll explore why techniques that work against traditional infrastructure break down once modern AI is integrated in an organization, setting the stage for the concepts and attack surfaces we'll build upon throughout the learning journey.

In this Learning Module, we will cover the following Learning Units:

We'll examine how AI behaviors, such as decision-making and autonomous actions, create new vulnerabilities and persistence paths. We'll review specific cases of these "new persistence paths" when we poison memory in agents and RAG pipelines in later Modules.

Finally, we'll introduce the AI Red Teaming Lifecycle and the key frameworks that guide our methodology throughout the course.

This Learning Unit covers the following Learning Objectives:

We've built this course for experienced penetration testers and red teamers who want to expand their skills to incorporate modern AI-enabled environments. We'll provide frameworks, mental models, and practical labs so we can identify, exploit, and report on AI-specific threats with confidence.

The course uses a modular, self-paced approach. Each Learning Module contains reading material, illustrative examples, and hands-on labs in contained environments. We'll leverage both on-premise machines and cloud-based environments to simulate real-world attack surfaces while keeping exercises contained and reproducible.

Each Module describes various classes of attacks against different components of AI-integrated environments, along with defensive tools and techniques. After completing all Modules, we'll tackle a Challenge Lab where we apply the course material in a large-scale simulated environment.

This course assumes familiarity with classic penetration testing techniques at the OSCP/PEN-200 level, including network enumeration, exploit development, host persistence, and evasion. It builds on that foundation by applying these skills to AI-specific targets such as models, retrievers, and agents. The LLM Red Teaming Learning Path, which covers the fundamentals of attacking LLMs, is strongly recommended.

By the end of the course, we'll be able to:

The course follows a deliberate progression from foundations through increasingly complex attack surfaces, culminating in a realistic engagement.

This is illustrated in the Course Roadmap below.

After establishing foundational concepts and frameworks in this Introduction Module, we move to Enumeration (Module 2) phase. Here we learn to identify AI-specific assets during reconnaissance, from public LLM deployments and RAG pipelines to agent frameworks, A2A protocols, and the AI model supply chain. We apply passive and active techniques while learning how AI telemetry differs from traditional

The heart of the course covers AI-Layer Attacks (Modules 3–7). We start with single-agent exploitation—prompt injection, memory poisoning, and tool manipulation, then scale to multi-agent systems where we impersonate agents, alter messages, and inject tasks into shared queues. We move through RAG pipeline exploitation (ingestion poisoning, retrieval hijacking, embedding attacks) and MCP tool surfaces (description poisoning, tool shadowing, parameter coercion), finishing with supply chain attacks targeting datasets, weights, LoRA adapters, and CI/CD pipelines.

We then pivot to Infrastructure Layer Attacks (Modules 8-9), targeting cloud ML services, model servers, and Kubernetes orchestration. We exploit misconfigurations that look benign until AI workloads are layered on top, and we cover adversarial ML techniques including model extraction and evasion.

Finally, the Synthesis phase (Modules 10-11) brings everything together. We formalize threat modeling for AI environments, mapping components to MITRE ATLAS and diagramming trust zones. The Capstone Red Teaming engagement is against MegacorpAI, a fictitious enterprise with multi-agent orchestration, RAG pipelines, MCP tools, and Kubernetes-hosted model servers—requires us to plan, execute, and report a full red team operation with blue-team telemetry enabled. During this capstone engagement, you may discover that "small" weaknesses from earlier Modules become the linchpin of your entire attack path.

The course culminates in a hands-on, 24-hour exam that requires the same set of skills as the Capstone Red Teaming engagement.

The Modules we described target a fundamentally different type of system than traditional red teaming engagements. To understand why we need specialized techniques, we must examine how AI systems differ from conventional infrastructure. Let's dive into the AI Security Landscape.

AI systems represent a fundamentally different class of security target.

This Learning Unit covers the following Learning Objectives:

The dynamic and generative nature of AI systems creates attack surfaces and business risks that traditional security models can't handle well.

Part of AI security involves attacking chatbots like OpenAI's ChatGPT or Anthropic's Claude. However, this is not our focus; we don't have permission to test these cloud services, and they're rarely in scope. The enterprise AI attack surface is broader: agent systems, RAG pipelines, model hosting platforms, supply chains, and orchestration frameworks.

When we compromise AI systems, we're not just stealing data, we're influencing decisions and actions. AI agents amplify both attacker capabilities and defensive challenges. Before we dive into attacks, let's establish some foundational concepts and frameworks.

AI is embedded in how organizations operate. According to the 2025 AI Index Report from Stanford HAI, U.S. private investment in AI reached $109.1 billion in 2024, with global generative AI investment hitting $33.9 billion. A 2025 METR study found that foundation model capability on software engineering tasks doubles every seven months.

This study shows that success rates on longer tasks, such as multi-step workflows, extended reasoning, and end-to-end processes, are increasing. Figure 5 illustrates that as reliability improves, businesses are more willing to delegate complex, multi-step work to AI systems. This shift reduces the marginal cost of completing complex tasks and increases overall productivity, making AI-driven workflows economically advantageous.

This capability growth directly appears in how organizations build software in areas such as code reviews and CI/CD automation.

Microsoft CEO Satya Nadella reported that 20–30% of Microsoft's code is AI-written. Google CEO Sundar Pichai said more than a quarter of new code is AI-generated. Microsoft CTO Kevin Scott predicted that within five years, 95% of all code will be AI-generated.

Even organizations claiming they don't use AI consume AI-generated code through vendor software, libraries, and frameworks.

This acceleration cuts both ways. AI serves as a mechanism for security (faster triage, anomaly detection, malware analysis) and an
object of security (high-value target with new attack surfaces). The same automation empowering defenders gives attackers more capability. If organizations don't secure AI systems now, attackers will gain ground faster than defenses can keep up.

In the context of LLMs, red teaming has expanded beyond traditional security concerns into what we might term safety: ensuring AI systems align with Responsible AI (RAI) principles, such as fairness, safety, privacy, and transparency.

[!warning]

RAI principles matter for red teamers because they define additional attack categories:

From a practical standpoint, we can patch a buffer overflow, but AI bias or ethical concerns are often subjective and context-dependent. Unlike traditional pentesting where attacks succeed or fail, many AI risks exist on a spectrum. An LLM might subtly frame answers misleadingly without producing outright misinformation. Red teaming for RAI requires multi-disciplinary expertise spanning security, AI ethics, social sciences, and philosophy.

Beyond these considerations, AI red teaming has profound technical consequences: it increases potential attacker impact. Models and agents make decisions, invoke tools, and generate outputs that humans and systems consume, including code deployed across organizations. If we can compromise the model, we alter those decisions at scale.

If we reliably manipulate an AI system, we may access sensitive data, escalate privileges, exfiltrate information, or mass-propagate malicious instructions. A single compromise can cascade into widespread operational, legal, and financial impact.

This changes three major things about how we approach AI red teaming.

First, value moves from files to behavior. Sensitive data no longer lives only in files or databases, but within model outputs and decisions. If we can leak embeddings, cause models to memorize training snippets, or manipulate responses, we expose critical information. We'll test how fragile this boundary is when we intentionally leak embeddings in later labs.

Second, persistence is dynamic. Traditional persistence uses disks and processes. In AI systems, we achieve persistence through poisoned datasets, vector database entries, or long-term memory edits. These survive container restarts, propagate into new deployments, and influence future outputs. A poisoned dataset or compromised component can remain effective even after redeployments or fine-tuning.

Finally, systems act autonomously. Agents call tools, open tickets, send emails, or trigger cloud actions. An attacker influencing an agent can automate large campaigns with minimal scaffolding. A single compromised prompt can generate thousands of malicious actions, as demonstrated in Anthropic's report on code execution risks in MCP.

Traditional red teaming focuses on accessing systems, establishing disk-based persistence, and demonstrating infrastructure control. AI red teaming targets behavior, long-term influence through poisoned data, and autonomous actions cascading across business operations. We're not just breaking into systems; we're manipulating decisions at scale.

AI compromises drain budgets, trigger regulatory action, and break trust in ways traditional breaches don't.

Gartner predicts that 40% of AI-related data breaches will come from cross-border generative AI misuse by 2027. IBM's 2025 Cost of a Data Breach report found the global average breach cost at $4.44 million, noting 16% of breaches now involve attackers using AI tools.

Our job as red teamers is expressing these risks in business terms by connecting technical exploitation to financial, regulatory, and operational consequences.

An attacker compromising an AI agent approving expense reports no longer needs to forge documents; they can manipulate the model to approve fraudulent transactions automatically. A single poisoned prompt or memory injection can cause thousands of approvals before detection. The loss is direct, measurable, and scales with automation.

Regulatory violations compound the damage. A poisoned document in a RAG system can subtly alter legal or compliance-related outputs, leading to violations in areas such as securities, healthcare, or data privacy. In regulated industries, a single compromised output—such as leaking patient data or approving unauthorized transactions—can trigger investigations costing millions. This risk is not limited to malicious actors. In practice, even legitimate users can introduce sensitive data into AI systems, as demonstrated in this article, where healthcare workers inadvertently exposed protected data through generative AI tools.

Reputational damage extends even further. When AI systems provide incorrect information or expose sensitive data, customers lose confidence, partners question the reliability of automation, and media coverage amplifies the impact. Organizations may take years to rebuild trust after high-profile AI failures.

Tesla’s Autopilot and Full Self-Driving systems illustrate this pattern. A series of fatal crashes triggered investigations by the National Highway Traffic Safety Administration (NHTSA), leading to recalls affecting millions of vehicles. Beyond regulatory consequences, sustained media coverage eroded public confidence in autonomous driving technology. The impact extended beyond Tesla, slowing adoption across the broader automotive industry.

In March 2026, NHTSA escalated a probe into Tesla's Full Self-Driving (Supervised) software after multiple crashes in low-visibility conditions such as fog, sun glare, and dust. The investigation advanced to an “engineering analysis,” a more serious stage that could lead to enforcement action, including a large-scale recall. The incident also contributed to a decline in Tesla’s stock price, highlighting the direct financial impact of AI system failures.

These examples demonstrate that failures in AI-driven systems are not isolated technical issues—they are business-impact events with cascading safety, financial, regulatory, and reputational consequences. To assess this impact systematically, we apply the following business impact questions:

Keep this checklist handy, as we'll reference it in the capstone when turning raw findings into board-ready impact statements.

These answers transform technical findings like "unvalidated RAG ingestion" into business risk statements like "Attackers can inject false contract terms into legal document retrieval, potentially causing regulatory violations carrying $500K in penalties and requiring three months of legal review." This impact analysis drives budget allocation and remediation priority.

As organizations adopt AI at scale, structured approaches for identifying and evaluating these risks are rapidly evolving.

To address AI security risks, three key frameworks have emerged: MITRE ATLAS, OWASP Top 10 for LLMs, and NVIDIA's AI Kill Chain.

Each serves a different purpose. MITRE ATLAS provides technique taxonomy. OWASP Top 10 identifies critical application-level risks. NVIDIA's AI Kill Chain maps the adversary lifecycle. Together, they form a comprehensive approach to AI security assessment.

These frameworks provide structured methodologies for assessing LLM security, standardized terminology for communication among security professionals, and globally-recognized best practices that align organizations with industry standards. They support collaboration across the AI security community, which is more valuable than any simple cheat sheet.

MITRE ATLAS extends the familiar ATT&CK framework into machine learning. It catalogs adversarial techniques across training, inference, and deployment, giving us taxonomy to tag behaviors consistently. When we exploit RAG systems in Module 5 or attack supply chains in Module 8, we'll reference corresponding ATLAS techniques. This helps defenders reference mitigations, structure playbooks, and standardize language across reports.

OWASP Top 10 for LLMs describes common application-layer problems: prompt injection, insecure output handling, RAG misconfigurations, and data leakage. For red teamers, OWASP provides categories for exploitation patterns to test. For blue teams, it offers targeted mitigations implementable in code or runtime policies.

NVIDIA's AI Kill Chain sequences attack activities through: Recon, Poison, Hijack, Persist, and Impact.

This reframes attacker goals in terms that map directly to AI artifacts and operations. We adopt this chain because it highlights where defenders can interrupt attacks, such as ingestion validation to prevent poisoning or tool vetting to prevent hijacking.

Module 2 applies the Recon stage. Modules 5 and 8 focus on Poisoning (ingestion and training manipulation). Hijacking appears in Modules 3 and 4 (prompt injection, immediate behavioral control). Persistence covers long-term influence via poisoned indices or memory. Impact ties everything together by measuring business effects.

ATLAS gives us shared vocabulary for documenting findings. OWASP provides application-level attack patterns to test. The NVIDIA Kill Chain helps sequence activities and communicate defensive gaps.

Before we move on, let's get familiar with the four different exercise formats we use throughout this course.

The first type is a knowledge check where we type the answer directly into the text field.

The second type is where we answer a multiple choice questions. We can enter either the option letter or the full text of the answer.

The third type is an open-ended question where we write a free-form response. There is no single correct answer and it is assessed whether the answer demonstrates understanding of the topic.

The fourth type is a hands-on lab exercise. We perform the desired action in the lab environment and then click the grade button.

Now that we understand the type of questions we will face, let's try two of them.

Using the business impact framework in Unit 1.2.3 from this module:

Log in using the following credentials: • Username: jamie.ross@megacorpai.com • Password: lab

After successfully logging in, interact with the assistant and ask it to explain the OWASP Top 10 for Large Language Models, then identify and list the top three risks.

In this Learning Module, we established why AI systems demand specialized security approaches. We explored how AI shifts value from static files to dynamic behavior, enables new persistence mechanisms through poisoned data and memory, and allows autonomous actions amplifying attacker impact. We examined business consequences including financial loss, regulatory violations, and reputational damage.

We then introduced three frameworks structuring our work: MITRE ATLAS for technique taxonomy, OWASP Top 10 for LLMs for application vulnerabilities, and NVIDIA's AI Kill Chain for sequencing attacks.

In the next Module, we'll apply these concepts to threat modeling and reconnaissance.