Malware downloaded from PyPI 41,000 times was surprisingly stealthy

reader comments

32 with 25 posters participating

PyPI—the open source repository that both large and small organizations use to download code libraries—was hosting 11 malicious packages that were downloaded more than 41,000 times in one of the latest reported such incidents threatening the software supply chain.

JFrog, a security firm that monitors PyPI and other repositories for malware, said the packages are notable for the lengths its developers took to camouflage their malicious code from network detection. Those lengths include a novel mechanism that uses what’s known as a reverse shell to proxy communications with control servers through the Fastly content distribution network. Another technique is DNS tunneling, something that JFrog said it had never seen before in malicious software uploaded to PyPI.

A powerful vector

“Package managers are a growing and powerful vector for the unintentional installation of malicious code, and as we discovered with these 11 new PyPI packages, attackers are getting more sophisticated in their approach, Shachar Menashe, senior director of JFrog research, wrote in an email. “The advanced evasion techniques used in these malware packages, such as novel exfiltration or even DNS tunneling (the first we’ve seen in packages uploaded to PyPI) signal a disturbing trend that attackers are becoming stealthier in their attacks on open source software.”

The researchers said that PyPI quickly removed all malicious packages once JFrog reported them.

Use of open source repositories to push malware dates back to at least 2016, when a college student uploaded malicious packages to PyPI, RubyGems, and npm. He gave the packages names that were similar to widely used packages already submitted by other users.

Over a span of several months, his imposter code was executed more than 45,000 times on more than 17,000 separate domains, and more than half the time, his code was given all-powerful administrative rights. Two of the affected domains ended in .mil, an indication that people inside the US military may have run his script.

been incorporated into multiple pieces of production software over a course of three months. Since then, there has been an almost countless number of instances of malware being sneaked into repositories.

In July, JFrog found malicious PyPI packages, downloaded more than 30,000 times, that carried out a range of nefarious activities, including stealing credit card data and injecting malicious code on infected machines. Earlier this year, a researcher developed a new type of supply chain attack that can have serious consequences. The so-called “dependency confusion attacks” work by uploading malicious packages to public code repositories and giving them names that are identical to legitimate packages stored in the internal repository of Microsoft, Apple, or another large software developer. Developers’ software-management apps often favor external code libraries over internal ones, so they download and use the malicious package rather than the trusted one.

From attacker to victim via Fastly

Now, these types of attacks are getting harder to detect. The biggest advance in subterfuge the researchers found was in two packages, one called “importantpackage” (or alternatively “important-package”) and the other called “10Cent10” (or “10Cent11”). The packages use the Fastly CDN to disguise communications between the infected machine and a control server.

The malicious code hiding in the packages causes an HTTPS request to be sent to pypi.python.org in a way that’s indistinguishable from a legitimate request to PyPI. The requests eventually are rerouted by Fastly as an HTTP request to the control server psec.forward.io.global.prod.fastly.net. The server then sends replies through the same setup, allowing for two-way communication. Fastly makes it easy for people to register their domains with the service. In many cases, registration can even be done anonymously.

JFrog researchers Andrey Polkovnychenko and Menashe explained:

The PyPI infrastructure is hosted on the Fastly CDN. This hosting uses the Varnish transparent HTTP proxy to cache the communication between clients and the backend. The traffic first goes into a TLS terminator for decryption, so the Varnish proxy can inspect the contents of the HTTP packet. The proxy analyzes the HTTP headers from the user’s request and redirects the request to the corresponding backend according to the Host header. The process then repeats itself in the reverse direction, allowing the malware to imitate duplex communication with PyPI.

As a result, the command-and-control (C2) session is encrypted and signed with a legitimate server certificate, making it indistinguishable from communicating with legitimate PyPI resources.

DNS tunneling, the other advanced evasion technique the researchers found, works using a DNS channel—normally reserved for mapping domain names to IP addresses—to send communications between an infected computer and a control server. DNS tunneling isn’t new, but the researchers said it’s the first time they have seen the technique used in malware uploaded to PyPI.

The growing sophistication of the malicious code being sneaked into PyPI, and presumably other repositories, is an indication that its use in spreading malware is likely to continue. Developers who rely on public repositories should take extra care to ensure there are no typos or stray letters in the package name they’re downloading.