The LiteLLM PyPI Compromise: Full Technical Teardown

Background

LiteLLM is one of the most widely deployed LLM gateway libraries in the Python ecosystem, with roughly 95 million downloads per month. It's built and maintained by BerriAI, a Y Combinator company, that provides a range of AI-powered tools and services for businesses and developers.

On March 24, two versions of LiteLLM were published to PyPI carrying a multi-stage credential stealer, an encrypted exfiltration channel, and a Kubernetes worm. The malicious packages were live on PyPI for several hours before being quarantined, though some mirrors and caches may have served the compromised versions for longer.

This was not a typosquat. It was not a rogue maintainer. It was the tail end of a campaign that began weeks earlier with a compromised security scanner, chained through CI/CD credential theft, and ended with attacker-controlled code landing on PyPI under a legitimate publisher identity.

We spent the past 24 hours pulling apart every layer of this incident. Here is what happened, how the payload works, and what you should be checking right now.

Timeline

The attack did not begin with LiteLLM. It began with the compromise of Trivy, a vulnerability scanning tool by Aqua Security.

Three tools compromised between March 19 and March 24: a container scanner (Trivy), an infrastructure-as-code scanner (KICS), and an LLM routing library (LiteLLM). All three are tools that, by design, run with elevated access to credentials and environment variables. That target selection is not random.

The Kill Chain

The initial foothold came through Trivy's GitHub Actions configuration. Trivy had a workflow called apidiff.yaml that used the pull_request_target event trigger. This trigger is dangerous because, unlike the regular pull_request event, it runs in the context of the base repository (not the fork). This means the workflow itself has access to the base repo's secrets and write permissions. If that workflow also checks out code from the incoming PR (i.e., the fork), an attacker can submit a malicious PR whose code executes with full access to those secrets. The attackers exploited exactly this pattern in late February to exfiltrate Aqua Security's aqua-bot credentials, which they then used to rewrite the Trivy v0.69.4 GitHub Action tags on March 19.

LiteLLM's CI/CD pipeline used the Trivy GitHub Action as part of its security scanning workflow, without pinning to a specific version. After the action was compromised on March 19, the poisoned action ran inside LiteLLM's build environment and exfiltrated the PYPI_PUBLISH token from the GitHub Actions runner.

With that token, the attackers had everything they needed to publish packages to PyPI under the legitimate LiteLLM publisher identity. No account takeover notification. No MFA prompt. The token was a pre-authorized secret sitting in the CI environment, and the poisoned Trivy action simply read it and sent it out.

This is worth pausing on. The compromise vector was not a phishing email or a leaked password. It was a CI/CD dependency — a security scanner, no less — that had itself been backdoored. The tool you run to find vulnerabilities became the vector for introducing one.

Krrish Dholakia, the CEO of BerriAI, also had his GitHub account compromised. This was later used to suppress the disclosure of the attack — flooding the issue with bot comments and closing it (detailed in the Suppression Attempt section below). However, the attackers used stolen credentials to directly push to LiteLLM's PyPI repository bypassing their usual review process.

Two Delivery Mechanisms

The attacker published two versions thirteen minutes apart, each escalating the delivery mechanism.

Version 1.82.7 embedded the payload directly in litellm/proxy/proxy_server.py, base64-encoded. It executes when anything imports litellm.proxy — the standard import path for LiteLLM's proxy server mode. This is the conservative approach: it fires when the application actually uses the library.

Version 1.82.8 retained the proxy_server.py payload from 1.82.7 and added a second, more aggressive mechanism: litellm_init.pth in site-packages/, a 34,628-byte double base64-encoded payload. The .pth mechanism is a CPython feature that fires on every Python interpreter startup — no import required. When pip runs, when your IDE's language server starts Python, when python -c "print('hello')" runs — the payload executes. CPython maintainers have acknowledged this risk in issue #113659, but no patch has landed.

The .pth file was correctly declared in the wheel's RECORD file with a matching hash. pip install --require-hashes would have passed. There is no integrity violation here — the malicious content was published using valid credentials. The cryptographic verification chain was unbroken. It just happened to be verifying attacker-controlled content.

This is a key distinction from most supply chain attacks people think about. Hash pinning, lockfiles, --require-hashes — none of these help when the publisher account itself is compromised. The package is "authentic" by every metric pip knows how to check.

The Payload

The malicious code is structured in three stages, each handling a distinct phase of the operation.

Stage 1 — Collection and Exfiltration

The first stage runs an extensive credential sweep. It is not subtle, and it is not narrow:

• System data: hostname, environment variables, running processes, network routing tables, recent entries from /var/log/auth.log
• SSH: private keys, known_hosts, authorized_keys, SSH agent sockets
• Git: credentials from ~/.gitconfig, credential helpers, .netrc
• Cloud: AWS credentials and config, GCP and Azure service account files, cloud instance metadata via IMDS (with full IMDSv2 signing implemented in the script), AWS Secrets Manager and SSM Parameter Store
• Container and orchestration: Docker config.json (registry credentials), Kubernetes kubeconfig, service account tokens, secrets, certificates
• Application secrets: .env files, Slack and Discord webhook tokens, CI/CD configs (Jenkins, Travis, Terraform), API keys from shell history
• Cryptocurrency: wallet files and seed phrases for Bitcoin, Ethereum, Solana, Cardano, Monero; Ledger hardware wallet configurations

The breadth here is notable. This is not a targeted operation looking for one specific credential type. This is a vacuum — everything gets swept up.

Collected data is encrypted with AES-256-CBC using a randomly generated 32-byte session key (via openssl rand). The session key itself is then encrypted with a hardcoded 4096-bit RSA public key (OAEP padding). The bundle is packaged into tpcp.tar.gz and exfiltrated via HTTPS POST. The two versions use different exfiltration endpoints: 1.82.7 sends to https://checkmarx[.]zone, while 1.82.8 sends to https://models.litellm[.]cloud/.

The models.litellm[.]cloud domain was registered on March 23, one day before the malicious packages went live. It is not affiliated with LiteLLM or BerriAI. The registrar is Spaceship, Inc., and the hosting provider is DEMENIN B.V. — the same infrastructure used across the Trivy and KICS operations. As of publication, there are no public reports of either models.litellm[.]cloud or checkmarx[.]zone being taken down or sinkholed.

Stage 2 — Persistence

A Python script is written to ~/.config/sysmon/sysmon.py and registered as a systemd user service at ~/.config/systemd/user/sysmon.service with the description "System Telemetry Service." It is enabled and started immediately.

The persistence script polls https://checkmarx[.]zone/raw every 50 minutes for a URL, downloads whatever is at that URL to /tmp/pglog, makes it executable, and runs it. At the time of analysis, the endpoint was returning YouTube links — a kill switch that causes the script to abort execution. The infrastructure is live and the operator can activate it by switching to real payload URLs whenever they choose.

Stage 3 — Kubernetes Lateral Movement

If the script finds a Kubernetes service account token at the standard in-cluster mount path, it reads all secrets across every namespace. It then attempts to deploy a privileged pod to every node in the cluster using alpine:latest, with the host filesystem mounted. These pods install the sysmon backdoor directly onto the underlying node.

The malicious pods are named node-setup-{node_name} (node name truncated to 35 characters) in the kube-system namespace, with a container named setup.

In a typical production Kubernetes environment, this means a single compromised pod with a service account token can cascade into full cluster compromise — every node backdoored, every secret exfiltrated.

The Discovery

Callum McMahon at FutureSearch was testing a Cursor MCP plugin that pulled in litellm as a transitive dependency. Shortly after Python started, his machine became unresponsive — RAM exhaustion. He traced it to the newly installed litellm package and found the litellm_init.pth file, 34,628 bytes, double base64-encoded.

The RAM exhaustion was an unintended side effect. Because the .pth mechanism fires on every Python interpreter startup, and the payload spawns a new Python subprocess, and that subprocess also triggers .pth execution, the result was a fork bomb. A bug in the malware, essentially. Without that bug, the payload would have run silently, and discovery would have been significantly delayed.

McMahon published his findings and opened GitHub issue #24512. Within two hours, the disclosure spread to Hacker News (324 points), r/LocalLLaMA, and r/Python.

The Suppression Attempt

When the disclosure issue went up, the attackers responded. Using the compromised krrishdholakia maintainer account, they:

1. Deployed 88 bot comments from 73 unique accounts in a 102-second window (12:44–12:46 UTC). The accounts were previously compromised developer accounts, not throwaway profiles. Rami McCarthy's analysis found 76% overlap with the botnet used during the Trivy disclosure.
2. Closed issue #24512 as "not planned."
3. Made commits to unrelated repositories with the message "teampcp update" — a deliberate signature.

The community routed around this by opening a parallel tracking issue (#24518) and continuing discussion on Hacker News.

About TeamPCP

The threat actor identifies as TeamPCP (aliases: PCPcat, Persy_PCP, ShellForce, DeadCatx3). They maintain Telegram channels at @Persy_PCP and @teampcp and embed the string "TeamPCP Cloud stealer" in their payloads.

The RSA public key hardcoded in the LiteLLM payload is identical to the one found in the Trivy and KICS payloads — the strongest single technical attribution link across all three operations. The tpcp.tar.gz bundle naming, tpcp-docs-prefixed GitHub repositories used as dead-drop C2 staging, and shared registrar/hosting infrastructure further connect the operations.

The LiteLLM compromise is Phase 09 of an ongoing campaign tracked by Wiz. TeamPCP has also deployed CanisterWorm, which uses the Internet Computer Protocol (ICP) as a C2 channel — canisters on ICP cannot be taken down by domain registrars or hosting providers. Aikido researchers have documented this as the first observed use of ICP for C2 in a supply chain operation, and have flagged a component called hackerbot-claw that uses an AI agent (openclaw) for automated attack targeting.

Blast Radius

The affected versions were on PyPI for an estimated two to five hours, depending on the source — Wiz recorded an initial quarantine shortly after discovery, while LiteLLM's own advisory lists a broader exposure window through 16:00 UTC. In that window, the following projects either pulled in the compromised versions or filed emergency PRs to pin away from them:

• DSPy (Stanford NLP) — CI failure triggered by the compromised dependency
• MLflow — emergency pin merged
• OpenHands — emergency pin merged
• CrewAI — decoupled from litellm entirely
• Arize Phoenix — emergency pin merged
• strands-agents/sdk-python — emergency pin merged
• langwatch, nanobot, dreadnode/rigging, CoPaw — all filed PRs or issues

The .pth mechanism means the payload can execute during CI/CD build steps. If any of these projects' CI pipelines ran pip install litellm without a version pin during that window, the build environment itself was compromised — not just the application.

Detection

If you touched litellm on March 24, here is what you should check.

Check your installed version:

pip show litellm | grep Version

If it shows 1.82.7 or 1.82.8, stop and treat the system as compromised.

Check for persistence artifacts:

ls -la ~/.config/sysmon/sysmon.py 2>/dev/null
ls -la ~/.config/systemd/user/sysmon.service 2>/dev/null
ls /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc /tmp/session.key.enc 2>/dev/null

Check for the .pth file:

find $(python3 -c "import site; print(' '.join(site.getsitepackages()))") \
  -name "litellm_init.pth" 2>/dev/null

Verify file hashes:

Check for network indicators:

grep -r "litellm\.cloud\|checkmarx\.zone" /etc/hosts /var/log/syslog 2>/dev/null

Check Kubernetes:

kubectl get pods -A | grep "node-setup-"

Remediation

If you had 1.82.7 or 1.82.8 installed, removing the package is not sufficient. The payload fires at install time. Your system may already be fully compromised.

Immediate steps:

1. Remove persistence: delete ~/.config/sysmon/sysmon.py, ~/.config/systemd/user/sysmon.service, and all temp files (/tmp/tpcp.tar.gz, /tmp/session.key, /tmp/payload.enc, /tmp/session.key.enc, /tmp/.pg_state, /tmp/pglog). Disable the systemd service.
2. Rotate every credential on the affected system. SSH keys, cloud access keys, API tokens, database passwords, Docker registry credentials, Kubernetes kubeconfig, service account tokens. If a credential existed on that machine — in memory, on disk, in an environment variable — assume it was read.
3. Audit AWS Secrets Manager and SSM Parameter Store if the affected system had IMDS access. The payload queries these directly.
4. Audit your Kubernetes cluster. If a service account token was present, all secrets across all namespaces may have been read. Check for node-setup-* pods in kube-system.
5. Rebuild affected systems from a clean state. Do not upgrade in-place.

If you were not affected:

Pin to litellm<=1.82.6 until a verified clean release is available. LiteLLM's maintainers have paused new releases pending a full supply chain review with Mandiant.

Takeaways

Hash verification does not protect against publisher compromise. pip install --require-hashes confirms a file matches what PyPI advertised. It does not tell you whether the advertised content is malicious. When the publisher account is the one that's compromised, the entire integrity chain is intact — it is just verifying the wrong thing.

CI/CD dependencies are part of your attack surface. LiteLLM was not directly targeted. Its CI/CD pipeline pulled in a compromised version of Trivy, which exfiltrated the PyPI publishing token. The blast radius of a CI tool compromise extends to every downstream project that trusts that tool.

The .pth mechanism is a pre-auth code execution primitive. Any package that can write a .pth file to site-packages/ gets code execution on every subsequent Python process startup, including pip itself. There is no opt-in, no import required, no user interaction. This has been a known issue for years.

Transitive dependencies are the real threat model. McMahon was not using LiteLLM directly. He was testing a Cursor MCP plugin that pulled it in as a transitive dependency. Most developers in the blast radius did not consciously choose to depend on LiteLLM — it arrived through their dependency graph.

Indicators of Compromise

Domains (not affiliated with LiteLLM or Checkmarx):

• models.litellm[.]cloud — exfiltration endpoint for 1.82.8 (POST)
• checkmarx[.]zone — exfiltration endpoint for 1.82.7 (POST), C2 polling for persistence (GET /raw)

Filesystem artifacts:

• ~/.config/sysmon/sysmon.py
• ~/.config/systemd/user/sysmon.service (description: "System Telemetry Service")
• /tmp/tpcp.tar.gz, /tmp/session.key, /tmp/payload.enc, /tmp/session.key.enc
• /tmp/.pg_state, /tmp/pglog
• litellm_init.pth in site-packages/

Kubernetes:

• Pods named node-setup-{node_name} in kube-system
• Container name: setup, image: alpine:latest

RSA public key prefix (identical across Trivy, KICS, and LiteLLM payloads):

MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvahaZDo8mucujrT15ry+...