Microsoft's GitHub under fire for DDoSing crucial open source project website

This month you may have noticed the servers used by the GMP project – an open source arithmetic library at the heart of GCC and other programs – slowed to a crawl. It was due to a deluge of network traffic, the source of which is quite surprising.

The packets appeared to come from servers associated with Microsoft.

Torbjörn Granlund, principal author of GMP, raised the alarm in a note to the project's mailing list.

"The GMP servers are under attack by several hundred IP addresses owned by Microsoft Corporation," he wrote. "We do not know if this is made with malice by Microsoft, if it is some sort of mistake, or if [it is one] of their cloud customers … running the attack. The attack targets the GMP repo, with thousands of identical requests. The requests are cleverly chosen as to cause heavy system load.

"We're firewalling off all of Microsoft's IP addresses as an emergency response."

The following day, Mike Blacker, director of threat hunting, operations, and response at Microsoft's GitHub, had identified the culprit: a GitHub Actions Workflow that clones a Mercurial repo and has been forked more than 700 of times.

"Microsoft and GitHub have investigated the issue and determined that a GitHub user updated a script within the FFmpeg-Builds project that pulled content from gmplib.org," explained Blacker.

"This build was configured to run parallel simultaneous tests on 100 different types of computers/architectures. This activity does not appear to be nefarious. [GMP] appears to have limited infrastructure that could not sustain the limited, yet simultaneous requests."

GitHub tries to prevent workflows from running on forked repositories. But the workflow clone defense doesn't work consistently.

• For a few days earlier this year, rogue GitHub apps could have hijacked countless repos
• GitHub publishes RSA SSH host keys by mistake, issues update
• Open source body quits GitHub, urges you to do the same
• GitHub accused of varying Copilot output to avoid copyright allegations

This is not the first time a software project has cried DDoS due to burdensome traffic demands. In February, 2022, Drew DeVault, founder of SourceHut, described the behavior of Google's Go Module Mirror as a distributed denial of service attack. After two years of complaints from DeVault, Google's Golang team earlier this year agreed to make its software less demanding on other people's computing resources.

Granlund was not entirely satisfied with Blacker's explanation, nor the implied feebleness of the project's server(s) – which, until a recent AMD Epyc 7402P upgrade, had been a not particularly robust Intel Xeon E5-1650 v2.

"Our machine is pretty powerful, it is a server class machine with many cores and lots of RAM, and its connection is 1GbE at a top-class datacenter," he replied.

"This is NOT a legitimate use of any server on the internet. Your reply seems to suggest that it is our fault, that we ought to have more powerful servers to accommodate this behavior. Really?"

That was Saturday, June 17, and Granlund fired off a subsequent missive to Blacker noting that the traffic flood remained ongoing and that he was continuing to block Microsoft addresses in response.

On June 18, the author of the FFmpeg-Builds published a commit to alert developers who fork the repository to adjust their workflow scripts. It checks the origin of the repo and, if not the original, echos a message to the developer's terminal:

As of last week, the excessive traffic was still an issue.

"Our servers are fully available again, but that's the result of us adding all participating Microsoft network ranges to our firewall," the GMP project explains on its webpage. "We understand that we are far from the first project to take such measures against Github."

They seem to think that they are entitled to bash away at smaller sites

The Register asked Granlund whether he was satisfied with Microsoft-GitHub’s response, and he told us he had only heard once from Blacker.

“I blocked about 40 IP ranges from accessing our web server,” he explained.

“A week after this started, there was still intensive traffic from the same IP addresses, perhaps 100 different Microsoft addresses all in all, belonging to about 40 ranges. The difference was that that traffic just caused minuscule load, and a log line in the firewall.

“Problem solved. I cannot care less if they no longer can access gmplib.org. I find it interesting how little responsibility GitHub-Microsoft assume here. They seem to think that they are entitled to bash away at smaller sites.”

GitHub did not immediately respond to a request for comment. ®

PS: If you noticed TLS-cert-issuing Let's Encrypt's hour-long outage this month, there's a technical analysis here by software engineer and cryptographer Andrew Ayer.