ENOSUCHBLOG

Programming, philosophy, pedaling.


Calculating my open source blast radius

Dec 12, 2021     Tags: oss, rant    


Preword

The health and stability of the open-source ecosystem is back in the ~discourse~, in no small part thanks to the ongoing nightmare with Log4j2 and CVE-2021-44228.

This post is my own contribution to that discourse, aimed at delivering a swift kick in the ass to anyone who thinks that we’re currently doing a good job of either (1) tracking and managing the tangled web of dependencies we’ve woven, or (2) supporting the maintainers whose un{,der}paid we all transitively depend on.

See also: Weird architectures weren’t supported to begin with.


Most programmers are familiar with the bus factor in software engineering: the number of engineers on a project who would need to have disaster befall them (being “hit by a bus”) in order to completely stop meaningful development.

For many open source projects, the bus factor is 1: a single maintainer performs the overwhelming majority of active development and support work, and controls the various credentials that distribute the project to the larger ecosystem1.

Even when projects develop formal governance (either independently, or via an umbrella group like the ASF) and collect multiple maintainers, the bus factor remains low: many people might hold commit or management permissions, but only a handful posses the overarching technical knowledge to make major changes or urgent, high-pressure fixes without doing more harm than good. Governance itself cannot solve this problem, because there’s only one thing that can: paying more open-source engineers enough to become experts on the systems they’re tasked with maintaining.

But these are staid observations, comprehensively argued by better engineers than me. So I’m going to focus on a related, but different metric: the blast radius for open source maintainers and their projects.

Open source blast radii

Open source projects exist on a spectrum of success2:

…and so forth, with everything in between.

The success of an open source project is only loosely tied to conventional engineering metrics: more engineers do necessarily not make a more popular project, nor does the “quality” or experience of those engineers guarantee success. Many projects (rightfully!) start as amateur experiments, and are picked up in good faith by the larger community because they solve real problems.

And so, we end up where we are today: we all depend on extraordinary amounts of open-source software, the quality (in terms of bugs, vulnerabilities, &c) of which is not strongly tied to how much we depend on it.

This is where I’d like to make my own ~infosec thoughtleader~6 contribution to the discourse: the concept of a blast radius for open source projects and their maintainers:

Open source blast radii have second-and-higher-order costs: they take time and money away from everyday engineering priorities (or even stop work altogether), require developers to rotate credentials and re-image their machines, and incentivize all kinds of well-intentioned but mostly useless corporate reactions10.

Measuring my own blast radius

Methodology

Using the definition above, I’m going to carve out a few conditions for my own projects:

The damage

ruby-macho

In my capacity as Homebrew’s least prolific and laziest maintainer, I maintain ruby-macho, the Ruby library that Homebrew uses to parse and rewrite every binary (executable, shared object, &c) that gets installed on macOS hosts. I’m proud of the work I did to develop ruby-macho, particularly because it requires so little maintenance: apart from dependency updates, it’s only needed 1-2 small changes a year to keep abreast with Apple’s changes to the Mach-O format.

At some point, the maintainers of CocoaPods (also written in Ruby) also needed Mach-O parsing, and adopted ruby-macho for their own purposes. By their metrics, CocoaPods is used to manage the dependencies in three million applications.

That brings us to:

A remotely exploitable vulnerability in ruby-macho would be a disaster: just about every engineer who develops on macOS would need to uninstall or at least avoid invoking Homebrew until a patch is available. Large parts of the Apple application ecosystem would need to be audited for incursion.

winchecksec

I helped write winchecksec as a side product of a funded research program at work. Like ruby-macho, it needs a relatively light touch: small updates whenever Microsoft adds another mitigation or relevant security metadata to the PE format, but nothing else.

At some point, $LARGE_ENTITY added winchecksec to the CI for $LARGE_PROGRAM14. I didn’t ask them to do that, but they did, and it’s still there for all I know. And that’s good! It’s a good tool, another one that I’m proud of. But once again my blast radius has grown: an exploit against winchecksec would potentially mean having to deliver client patches to hundreds of millions of users across every mainstream operating system. Not an enviable task.

Other bits

I excluded from consideration some of the largest projects I’ve worked on recently, primarily because their bus factors are thankfully higher than 1 (in some cases, much higher).

But that alone doesn’t mean that those projects are adequately funded or healthy along any particular metric15, or even that those projects don’t have large blast radii. It only demonstrates that my own blast radius smaller than the sum of the radii of all the projects I help maintain, a fact that I’m eternally thankful for.

This points to a useful metric to optimize for: some projects will always have large blast radii16 but, in an ideal world, every maintainer would have a personal blast radius of 0.

Narrowing the blast radius

Right now, all is calm: to the best of my knowledge, nobody is trying to turn my open source work into a liability for the projects and engineers who depend on it. But that doesn’t change the threat.

What can I (or anyone else) do to improve the situation? I can think of a handful of things:

With each of these steps, the blast radius of each project gets a little bit smaller: even in the event of my total vaporization by a sufficiently high-speed bus disappearance, another maintainer could step in, get a feel for the territory using my notes, and release a version with a fix in a timely manner. The vulnerability might still happen, but we can cut back the time on the fix and save everyone a whole bunch of time and money in the process.

I’ve tried my best to do some of these. But it should come as no surprise that some (all?) of them take time17, and you know what they say about time.

Wrapup

I beat myself to the punch a little bit on the Blue Bird Site:

We are not going to make meaningful progress on open-source sustainability until companies start running the books and tallying up the money they’re losing by failing to pay for the open-source labor they rely on.

Many current corporate incentives run counter to a positive change here: companies see their incident response and blue teams as a fixed, sunk cost that they might as well get some value out of when the shit hits the fan18. What they don’t seem to see is the knock-on effects mentioned above: the hours-turned-days-turned weeks lost each time a work-stopping exploit happens, to say nothing of the avalanche of policy and procedural changes that follow every incident.

I’m under no delusion I am a particularly important player in the open source world, or that the projects I’ve listed above deserve special attention19. The point is in fact the opposite: I am a relative nobody, and my work still ends up on the critical path for millions of people. And that’s business as usual!

This post ended up being more of a ramble than I was hoping. To tie a bow around things:


  1. Continuous integration access, GitHub organization management, release tokens, codesigning certificates, &c. 

  2. i.e. popularity, adoption, whatever. 

  3. Probably the overwhelming majority, if we consider a “project” to be something like a repository on GitHub. 

  4. Limited public applicability being a large and reasonable one. 

  5. Nix is also a fantastic example of this. 

  6. Feel free to bury me on the day I become one. 

  7. Measured in money and/or money equivalents, like engineer-hours. 

  8. Maybe this should be higher? I don’t know; this is a first blush. 

  9. There is, of course, a great deal of wiggle room with what “sustainably funded” means for any given open source project; it can be anywhere from $100 and free CI credits to a multi-million-dollar foundation budget and multiple in-person all-expenses-paid developers’ summits. This alone requires far more rigorous attention than we’ve given it. 

  10. More endpoint security software! Pay for another SOC 2 vendor! Make every engineer pinky promise not to introduce vulnerabilities! 

  11. I don’t know the exact number. 

  12. curl https://formulae.brew.sh/api/analytics/install/30d.json | jq -r '.items | .[] | .count' | tr -d ',' | awk '{s+=$1} END {print s}' 

  13. I suspect I’ll never know exactly how many. That being said: CapitalOne, Google, Stripe, Square, Slack, and a number of other very large companies appear to sponsor CocoaPods, strongly indicating that it (and thus, transitively, me) are part of their development ecosystem. 

  14. Names have been removed not because it’s a secret or shameful, but because it doesn’t add anything to the story. They haven’t done a single thing wrong. 

  15. Some good metrics being whether maintainers are de facto “on call” for companies they never agreed to work for, whether they are chained to their projects professionally, &c. 

  16. We aren’t going to (and shouldn’t) get rid of libc, the Linux kernel, OpenSSL, &c, and memory safety alone won’t save us. 

  17. Either my time or others. 

  18. And don’t read this as an anti-IR or blue-team screed: those teams are vital! No amount maintainer compensation will perfectly eliminate vulnerabilities or the need for talented individuals on staff to perform response. 

  19. It’s also not an oblique demand for payment. I am already paid to work on open source. I want others to be paid.