The Escalating Threat of Software Supply Chain Attacks: TeamPCP Breaches `litellm`
In a stark reminder of the ever-present dangers in the open-source ecosystem, the notorious threat actor TeamPCP has once again sent ripples through the cyber security landscape. Following previous compromises of critical tools like Trivy and KICS, TeamPCP has now infiltrated the popular Python package `litellm`, deploying malicious versions that incorporate a credential harvester, a potent Kubernetes lateral movement toolkit, and a persistent backdoor. This incident underscores a worrying escalation in software supply chain attacks, directly impacting developers and organizations leveraging these crucial open-source components. Dive in to understand the sophisticated multi-stage payload and the urgent actions required to protect your infrastructure.
TeamPCP’s Latest Strike: Compromising the `litellm` Python Package
TeamPCP, a threat actor with a disturbing track record, has escalated its campaign by compromising the `litellm` Python package. This move follows their previous successful incursions into security tools like Trivy and KICS, indicating a clear strategy to target high-leverage points within the development lifecycle. Security vendors, including Endor Labs and JFrog, confirmed that malicious versions 1.82.7 and 1.82.8 of `litellm` were published on March 24, 2026, and have since been removed from PyPI. The incident is believed to be a direct consequence of `litellm`’s reliance on Trivy in its CI/CD workflow, highlighting the cascading risk inherent in complex software dependencies.
Unpacking the Malicious Payload: A Three-Stage Threat
The payload delivered within the compromised `litellm` versions is a sophisticated, multi-stage attack designed for maximum impact:
- Credential Harvester: This component is engineered to sweep a wide array of sensitive data, including SSH keys, cloud credentials, Kubernetes secrets, cryptocurrency wallets, and `.env` files.
- Kubernetes Lateral Movement Toolkit: A highly dangerous capability that deploys privileged pods to every node within a compromised Kubernetes cluster, enabling extensive control and reconnaissance.
- Persistent Backdoor: Implemented as a systemd backdoor (named `sysmon.service`), this ensures continuous access by polling `checkmarx[.]zone/raw` for additional binaries.
All harvested data is exfiltrated as an encrypted archive (`tpcp.tar.gz`) to the command-and-control domain `models.litellm[.]cloud` via HTTPS POST requests, making detection challenging without deep network monitoring.
Evolution of the Attack Vector: From Import to Persistent Execution
The initial malicious version, 1.82.7, embedded its harmful code within the `litellm/proxy/proxy_server.py` file. The injection occurred during or after the wheel build process, ensuring execution at module import time. This means any process importing `litellm.proxy.proxy_server` would unknowingly trigger the payload without requiring any explicit user interaction.
Version 1.82.8 introduced an even more aggressive vector: a malicious `litellm_init.pth` file placed at the wheel root. Python’s `.pth` files are automatically processed by `site.py` during interpreter startup. This critical distinction means the malicious logic would execute automatically on *every* Python process startup in the environment, not just when `litellm` is specifically imported. Furthermore, this iteration spawns a child Python process via `subprocess.Popen`, allowing the payload to run persistently and discreetly in the background, making it harder to detect and terminate.
The payload orchestrator, once executed, unpacks the credential harvester and persistence dropper. It then leverages any available Kubernetes security context, such as service account tokens, to enumerate cluster nodes and deploy highly privileged pods. These pods then `chroot` into the host filesystem to install the systemd user service (`~/.config/sysmon/sysmon.py`) for persistent access. This script periodically contacts `checkmarx[.]zone/raw` for next-stage payloads, notably including a `youtube[.]com` URL as a kill switch—a pattern observed across TeamPCP’s previous attacks.
TeamPCP’s Relentless Campaign: A Threat to Open Source Ecosystems
TeamPCP’s campaign is far from over. Their consistent pattern of leveraging compromised environments to gain credentials for their next target represents a deliberate and dangerous escalation. The pivot from CI/CD environments (GitHub Actions runners) to production-grade PyPI packages running in Kubernetes clusters demonstrates a strategic expansion of their targeting footprint.
This relentless campaign has now spanned five critical open-source ecosystems: GitHub Actions, Docker Hub, npm, Open VSX, and PyPI. Socket highlighted that TeamPCP is “escalating a coordinated campaign targeting security tools and open source developer infrastructure, and is now openly taking credit for multiple follow-on attacks across ecosystems.”
TeamPCP themselves have openly boasted on their Telegram channel, stating, “These companies were built to protect your supply chains yet they can’t even protect their own, the state of modern security research is a joke, as a result we’re gonna be around for a long time stealing terrabytes [sic] of trade secrets with our new partners.” They further warned of a “snowball effect” and future collaborations to “perpetuate the chaos,” with “many of your favourite security tools and open-source projects” targeted in the months to come.
Gal Nagli, head of threat exposure at Google-owned Wiz, succinctly summarized the gravity of the situation: “The open source supply chain is collapsing in on itself. Trivy gets compromised → LiteLLM gets compromised → credentials from tens of thousands of environments end up in attacker hands → and those credentials lead to the next compromise. We are stuck in a loop.”
Immediate Actions to Mitigate the Threat
Given the severity and breadth of this attack, immediate and thorough action is crucial to contain the threat and prevent further compromise. Organizations and developers should undertake the following:
- Audit and Revert: Scan all environments for `litellm` versions 1.82.7 or 1.82.8. If found, immediately revert to a clean, trusted version.
- Isolate Affected Hosts: Disconnect or quarantine any systems identified with the malicious packages to prevent lateral movement.
- Check Kubernetes Clusters: Scrutinize Kubernetes clusters for the presence of rogue pods that may have been deployed by the lateral movement toolkit.
- Review Network Logs: Analyze network egress traffic for any connections to `models.litellm[.]cloud` and `checkmarx[.]zone`, which indicate C2 communication.
- Remove Persistence: Identify and remove any installed persistence mechanisms, specifically the `sysmon.service` backdoor.
- Audit CI/CD Pipelines: Review CI/CD pipelines for any usage of tools like Trivy and KICS during the periods of their known compromise, as this could be an entry point.
- Revoke and Rotate Credentials: Critically, revoke and rotate all credentials that may have been exposed or harvested, including API keys, cloud access keys, and SSH keys.
FAQ
Question 1: What makes TeamPCP’s supply chain attacks particularly sophisticated and dangerous?
TeamPCP’s attacks are sophisticated due to their multi-stage nature, combining credential harvesting, lateral movement, and persistent backdoors. They are dangerous because they target high-leverage points in the software development lifecycle, like CI/CD tools and popular open-source packages. By compromising tools developers trust, they achieve deep access into environments, moving from CI/CD to production systems and exfiltrating sensitive data like cloud credentials and Kubernetes secrets. Their use of .pth files for persistent, stealthy execution further amplifies the risk.
Question 2: What immediate technical steps should my organization take if we’ve used litellm?
If you’ve used litellm, immediately audit all environments to check for versions 1.82.7 or 1.82.8. If detected, isolate affected hosts, revert to a known clean version of litellm, and scrutinize Kubernetes clusters for rogue pods. Crucially, review network logs for any egress traffic to models.litellm[.]cloud or checkmarx[.]zone and remove any detected persistence mechanisms like sysmon.service. Finally, revoke and rotate all potentially exposed credentials as a priority.
Question 3: Beyond immediate remediation, what long-term strategies are crucial for bolstering software supply chain security against such advanced threats?
For robust long-term software supply chain security, organizations must implement a multi-layered approach. This includes adopting Software Composition Analysis (SCA) and dependency scanning tools, enforcing strict access controls with multi-factor authentication (MFA) for all package repositories, and employing runtime application self-protection (RASP). Additionally, consider integrating artifact signing (e.g., Sigstore) to verify the integrity and origin of software components, and apply a ‘zero-trust’ mindset to all third-party dependencies. Regularly auditing CI/CD pipelines for vulnerabilities and anomalous activity is also essential to break the attack chain.