The last release (the 4.x series) was exactly 2 years ago, so it’s been a little while.

Big new things

This release series has two primary features:

• ff2mpv now supports configurable profiles, which can be created and modified directly from the extension. The associated pull request has all of the relevant details, but the relevant bit is this (under about:addons > ff2mpv > Preferences):

  

  As the big scary message indicates, you must be careful about what you put in your MPV profiles: these are arbitrary flags that are passed to the underlying mpv invocation, and MPV has ample ways to execute arbitrary code. Caveat emptor.

  Many thanks to @DanSM-5 for his development efforts on this feature.

• External extensions can now invoke ff2mpv via the browser.runtime.onMessageExternal API, by user demand. The change itself was small (just 23 additional lines), but will apparently make ff2mpv compose with other extensions.

  Thanks again to @DanSM-5 for this feature enhancement as well.

Smaller new things

• Outside of the extension itself, support for other browsers (beyond Firefox and mainline Chrome/Chromium) has continued to improve. Or at least I think it has, because people only sporadically file bugs for Brave, LibreWolf, &c. Some relevant changes over the last two years:

  • Support for more browsers in the installation scripts by @eNV25
  • An install script for Windows by @DanSM-5
  • A new Rust native host implementation by @ryze312
  • Support for Brave in the Windows installation script by @DanSM-5
  • A fix for installations on arm64 macOS by @claudiofreitas
  • …and a whole bunch of miscellaneous documentation fixes and improvements by various contributors. Thank you all!

Upcoming things

ff2mpv continues to be very conservatively developed: support for MPV profiles is the biggest feature addition in years, and is likely to remain the biggest for a long time.

The biggest upcoming thing is the move to Manifest V3, which Chrome is making us do. There’s a work in progress PR for it, but we still have a few months until anything needs to change (provided Chrome doesn’t push the migration back again).

When MV3 comes out, I will likely begin the 6.x series. Until then, enjoy the 5.x series and please report any bugs you run into!

I think most (personal) blogs don’t need browser analytics¹. This blog has no browser analytics; I removed them in back in 2017, with no discernible negative impact on my writing motivation, my ability to observe general traffic trends, or the blog’s popularity.

TL;DR for everything below: the only way I track traffic trends on this blog is with vbnla, a hacky little Ruby script that summarizes my Nginx access log. This has been more than sufficient for assessing post popularity and the general geographic and link origins for incoming traffic.

Writing motivation

I originally added analytics (in the form of Google Analytics) to the blog back in 2015, when I was still in college.

My original rationale for adding analytics was to to motivate myself to write posts on a regular schedule: I thought that having a pretty dashboard of (ever increasing, in my natural human desire for attention) visitors would induce additional writing from me each month.

The opposite turned out to be true: like most blogs, nobody really read mine, and my visibility into that demotivated me and made me feel like giving up on a blog entirely. Looking back, this was inevitable, but having analytics made it worse: instead of only evaluating my own desire to write and improve my blogging, I became (resentfully) dependent on a small trickle of statistics.

Ultimately, there’s nothing wrong with deriving some motivation from the attention that can come with blogging (I still do, as noted below). But it’s also the cheapest source of motivation, in the senses of being the easiest to track, least reliable, and ultimately not tied to a passion for the act of writing itself. Turning off the blog’s browser analytics helped me decrease the overall degree of motivation that came from external attention, while increasing the overall degree I felt from both habituation (writing at least one post a month) and post quality (feeling that I was actually expressing something interesting to myself, rather than attempting to track readers’ interests).

Analytics when I need them

Like any good blog post title, there is a kernel of untruth in this one: analytics are useful for debugging purposes², tracking overall server load³ and, more selfishly, understanding when a particular post is getting attention.

I still have a little bit of insight into my blog’s traffic for these things, via my web server’s access log. Each log entry looks like this:

1
XXX.YYY.AAA.BBB - - [24/Dec/2023:00:10:20 -0500] "GET /2022/12/28/ReDoS-vulnerabilities-and-misaligned-incentives HTTP/1.1" 200 9654 "https://some-referrer.example.com" "Mozilla/5.0"

(where XXX.YYY.AAA.BBB is the IP address that made the request).

This tells me just about everything I could need to know for the aforementioned debugging, load monitoring, and selfish attention purposes. It also gives me essentially the same things that I would want from browser analytics, without injecting third-party code into each reader’s browser:

• IP addresses give me rough geolocation information, for understanding where readers are coming from (or sources of significant automated traffic, like RSS pollers and search crawlers);
• User agents and the HTTP Referer give me rough readership demographics;
• The accessed URL tells me what’s actually being read.

In practice, I rarely use this information. But it does exist, and it gives me everything that I would have wanted from browser analytics.

When I do use it, I use it through vbnla, a hacky little script that collates the log’s recorded requests by top IPs, user agents, requests, and referrers. Here’s what that looks like on today’s log, as of this morning (with IPs redacted):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
Total requests: 5195
Unique IPs: 878
Top 10 IPs:
	REDACTED -> 160
	REDACTED -> 92
	REDACTED -> 72
	REDACTED -> 71
	REDACTED -> 63
	REDACTED -> 52
	REDACTED -> 51
	REDACTED -> 50
	REDACTED -> 49
	REDACTED -> 47

Top 10 UAs:
	NetNewsWire (RSS Reader; https://netnewswire.com/) -> 241
	FreshRSS/1.22.1 (Linux; https://freshrss.org) -> 223
	- -> 200
	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 -> 159
	Yarr/1.0 -> 121
	Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0 -> 96
	Readdigbot (+https://www.readdig.com) -> 92
	FreshRSS/1.21.0 (Linux; https://freshrss.org) -> 90
	Bifolia:0.1.1 -> 71
	Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0 -> 70

Top 10 requests:
	/.env -> 22
	/2020/12/24/A-few-HiDPI-tricks-for-Linux -> 20
	/2020/09/19/LLVMs-getelementptr-by-example -> 12
	/2021/07/19/LLVM-internals-part-1-bitcode-format -> 11
	/2022/02/02/Setting-up-Navidrome-with-Nginx-as-a-reverse-proxy -> 8
	/2023/10/18/Some-concerns-with-OpenPubKey -> 8
	/login.php?s=Admin/login -> 8
	/2023/09/22/GitHub-Actions-could-be-so-much-better -> 8
	/2020/11/30/How-many-registers-does-an-x86-64-cpu-have -> 7
	/2020/12/16/Static-calls-in-Linux-5-10 -> 6

Top 10 referrers:
	- -> 4522
	https://blog.yossarian.net/feed.xml -> 443
	https://blog.yossarian.net/ -> 59
	https://www.google.com/ -> 22
	https://blog-yossarian-net.translate.goog/ -> 16
	http://blog.yossarian.net/feed.xml -> 12
	https://yandex.ru/ -> 7
	https://duckduckgo.com/ -> 6
	http://yossarian.net -> 5
	https://yossarian.net/docs/libmsr/ -> 3

This is what using upgrayedd looks like:

1
2
3
4
5
6
7
8
use upgrayedd::upgrayedd;

#[upgrayedd]
fn X509_VERIFY_PARAM_set_auth_level(param: *mut std::ffi::c_void, level: std::ffi::c_int) {
    eprintln!("before!");
    unsafe { upgrayedd(param, level) };
    eprintln!("after!");
}

If you build that in a crate with crate-type = ["cdylib"], then you can do this:

1
2
3
# libfunkycrypto is whatever your crate's shared object build target is
LD_PRELOAD=./libfunkycrypto.so \
    curl --silent https://example.com --ciphers DEFAULT@SECLEVEL=2

…to run custom code before and after each call to OpenSSL’s X509_VERIFY_PARAM_set_auth_level.

The rest of this post will be a brief introduction to (dynamic) function interposition and how it works, upgrayedd’s implementation details, and what it can (and can’t) be used for.

(Dynamic) function interposition

Function interposition is a basic program instrumentation technique: to measure, detect, or modify the use of a function in a program, we replace calls to that function with calls to a function that we (the instrumenter) control. The interposed (“wrapper”) function can then monitor (and rewrite) the program’s state, including:

• Changing the parameters passed into the target (“wrapped”) function, causing it to behave differently;
• Changing the return value from the target, causing the caller to behave differently;
• Skipping the target entirely (stubbing or emulating it);
• Modifying the program’s larger state, including globals, handles on system resources, &c.

There are many different ways to interpose on a program’s functions, but one of the simplest is the LD_PRELOAD trick:¹ when set, the dynamic linker/loader will give precedence to the symbols defined in the specified shared object.

The wrapper can then access the underlying replaced function via real_func = dlsym(RTLD_NEXT, ...), where RTLD_NEXT is a special pseudo-handle that tells dlsym(3) to retrieve the next occurrence of the symbol in the dynamic linker’s search order.

In practice, this means that any dynamic² function call can be interposed with LD_PRELOAD, including “basic” routines like malloc³.

How upgrayedd works

As hinted above: upgrayedd works through LD_PRELOAD, which isn’t that special.

What does make upgrayedd special is its ability to abstract much of the (error-prone) boilerplate that comes with writing function instrumentation and interposition code.

Compare, for example, the following upgrayedd wrapper on rand(3):

1
2
3
4
5
6
7
use upgrayedd::upgrayedd;

#[upgrayedd(real_rand)]
fn rand() -> std::ffi::c_int {
    // The domain of [0, 42) is random enough.
    (unsafe { real_rand() }) % 42
}

…with its rough equivalent in C:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#include <stddef.h>

#define _GNU_SOURCE  // for RTLD_NEXT
#include <dlfcn.h>

static int (*_real_rand)() = NULL;

int rand() {
  if (!_real_rand) {
    _real_rand = dlsym(RTLD_NEXT, "rand");
  }

  // The domain of [0, 42) is random enough.
  return _real_rand() % 42;
}

The C version is slightly more verbose and significantly more error prone: the function signature is written twice (once for the wrapper, and once for the function pointer containing the target), and the C compiler won’t catch any errors in either.

A naive Rust implementation would have similar problems; upgrayedd mostly sidesteps these by abstracting away each individual step behind its procedural macro.

When expanded, the rand example above looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
static mut __upgrayedd_target_rand: Option<unsafe extern "C" fn() -> std::ffi::c_int> = None;

#[no_mangle]
#[doc(hidden)]
#[allow(non_snake_case)]
#[export_name = "rand"]
pub unsafe extern "C" fn __upgrayedd_inner_wrapper_rand() -> std::ffi::c_int {
    if __upgrayedd_target_rand.is_none() {
        __upgrayedd_target_rand = std::mem::transmute(
            ::libc::dlsym(::libc::RTLD_NEXT, std::mem::transmute(b"rand\x00".as_ptr())),
        );
    }
    if __upgrayedd_target_rand.is_none() {
        let msg = b"barf: upgrayedd tried to hook something that broke rust's runtime: ";
        ::libc::write(
            ::libc::STDERR_FILENO,
            msg.as_ptr() as *const ::libc::c_void,
            msg.len(),
        );
        ::libc::write(
            ::libc::STDERR_FILENO,
            b"rand".as_ptr() as *const ::libc::c_void,
            b"rand".len(),
        );
        ::libc::write(::libc::STDERR_FILENO, b"\n".as_ptr() as *const ::libc::c_void, 1);
        std::process::abort();
    }
    rand()
}

#[allow(non_snake_case)]
fn rand() -> std::ffi::c_int {
    #[allow(unused_variables)]
    let real_rand = unsafe { __upgrayedd_target_rand.unwrap_unchecked() };
    (unsafe { real_rand() }) % 42
}

…which is a mess to read, but is essentially the same thing as the C version, but with a few extra checks (most notably, a hard abort if dlsym fails to retrieve the target function⁴).

Limitations

upgrayedd comes with a few caveats beyond the normal limitations of LD_PRELOAD:

• For the time being, it uses Rust’s std (and more generally, users may somewhat reasonably expect to be able to use crates or logic that needs std in their interpositioned code). As a result, upgrayedd may not always hook where you expect if mingled inside of a larger Rust codebase.

  Bottom line: use it in small codebases, with minimal dependencies⁵.

• Despite being written in Rust (tm) and exposing a “safe” wrapper (until you choose to call the underlying target), upgrayedd is fundamentally unsafe and cannot be made safe: interposition fundamentally involves unsafe castings of pointers behind symbols, with no guarantee that the signature is correct (or that the symbol is even a function⁶).

  Even just declaring an upgrayedd hook can cause (silent!) memory corruption if the type or signature behind the symbol is not the expected one. You should not use it in anything that needs to be stable or secure; its primary value is in writing instrumentation tooling that stays on a developer’s workbench.

• It’s Linux-only for the moment, for the simple reason of “I haven’t bothered to make it work anywhere else.” It would probably work well on the BSDs with minimal changes, and should work on macOS with SIP disabled; I don’t know if Windows supports this kind of thing⁷.

Wrapup

Ultimately, upgrayedd is a bikeshed/waypoint towards a larger tool that I’m working on for detecting API misuse. Stay tuned for more news on that.

Despite writing Rust for ~5 years now, this was my first time creating (rather than just modifying) a proc-macro crate.

Developing a procedural macro was an interesting experience: being able to control the generated syntax was both very freeing and also very constraining (the complexity of Rust’s AST is on full display in syn’s APIs, which makes even “trivial” looking transformations⁸ somewhat complicated). I once again found myself wishing for a macro system like Crystal’s, where a small amount of flexibility is exchanged for substantially simpler “template” style transformations.

This post was mostly written from an airplane. Please forgive any typos or small errors; I appreciate being notified of them and will make an effort to fix them.

Disclosure: I am professionally involved in the Sigstore community, and currently maintain Sigstore’s Python client. While I work on Sigstore professionally, the content below is my personal opinion and not necessarily the opinion of my employer, the Sigstore project, or anyone else in this general space.

Earlier this month, the Linux Foundation announced OpenPubKey as a new member project. You can read their announcement here. Some other resources, which have been around for a few weeks to months, came to my attention at around the same time:

• OpenPubKey’s reference implementation
• A pre-print on IACR ePrint for OpenPubKey
• BastionZero’s OpenPubKey explainer and blog post

I’ll start by saying that I think the idea behind OpenPubKey is extremely cool and demonstrates the (basic) workability of a technique (binding an ephemeral signing key to a semi-permanent identity in a globally verifiable way without additional trusted services) that I think is both extremely useful and powerful.

At the same time, I have concerns about how OpenPubKey’s privacy properties, its actual ability to provide reliable “keyless” signatures, and its compatibility with and implications for OIDC practices within IdPs. This post is an attempt to elaborate on those concerns.

Finally: I don’t believe these concerns are uniquely mine. There’s a post on the Sigstore blog that in particular covers some of the same privacy concerns, and probably others that I’ve missed.

Quick background

At a very high level: Sigstore and OpenPubKey attempt to solve the same problem, namely: giving users the ability to produce signatures bound to a publicly verifiable identity. Doing this reliably solves two huge problems in “ordinary” distributed¹ signing application:

1. Identity establishment. Traditional signing schemes punt identity verification to the end user, e.g. by expecting them to confirm that a certificate claiming to represent an entity is in fact controlled by that entity.

   Binding key material directly to an identity credential sidesteps this: the user still needs to determine if they trust a particular identity, but an attacker can no longer easily² impersonate that identity with key material that they wholly control.

2. Key management. Even when traditional signing schemes produce identity mappings that they are confident in, they must still perform manual key management: rotating out expired keys, periodically retrieving new keys for identities (and verifying them either in-band or out-of-band), &c. These tasks are manual, operationally complex, and error prone.

   Using verifiable identities rather than the keys themselves sidesteps all of this and allows all parties to effectively be “keyless” at the end signature level: all private key material for signing operations can be discarded immediately, since binding a new ephemeral key to the same identity is relatively inexpensive.

Both Sigstore and OpenPubKey used OpenID Connect as their underlying source of identity and identity credentials, but their use of those credentials differs significantly:

• Sigstore maintains a set of services, which it calls the “public good” instance. These services include a Certificate Authority (Fulcio) that’s responsible for accepting identity tokens, binding them to signing keys, and publishing the bound result as an X.509 certificate.

  Sigstore also includes transparency services (a CT log for Fulcio itself, plus Rekor for artifact transparency) that are intended to keep Sigstore’s infrastructure auditable and honest, but using Sigstore does fundamentally involve placing trust in its CA.

• OpenPubKey does not use a Certificate Authority, or any other external services besides the identity provider itself. Instead, it performs a trick (more on this in a bit) to bind a signing key directly to the identity credential produced by the identity provider.

Or, as a TL;DR: Sigstore’s design involves trusted (but transparent and publicly auditable) services that transform an identity credential and a public key into a new certificate, while OpenPubKey’s design involves producing an identity credential with a public key already bound to it.

Privacy

As mentioned, OpenPubKey does not use a Certificate Authority or other form of indirection to bind a signing key to an identity token. Instead, the identity token itself is generated by the identity provider in a way that binds it to the key, and is directly shared with verifying parties (including potentially the public Internet).

This presents a problem: OIDC is intended to be a private credential format, with the expectation that only a few designated parties³ are trusted to receive and verify it.

Consequently, OIDC identity providers may include default claim values in their identity tokens that are neither required nor expected in a public key binding: things like secondary email addresses, service account identifiers, account statuses (e.g. whether using 2FA, whether suspended), gender, birth date, &c. OpenID Connect even includes a standard claim for mailing addresses!

More generally (and this will be a recurring theme), OpenID Connect providers are not bound to only exposing a particular set of claims over time: providers may (and, in practice, regularly do) choose to change their claim sets and claim contents over time. In other words: an identity credential that previously only exposed an email address could suddenly being exposing DOB and gender claims, with no advance notice (or particular visibility, unless the user is manually inspecting the JWT claims on each signing operation).

At best, this is extremely surprising: OpenPubKey users may believe that they’re only exposing a “sufficient” amount of identity information for verification purposes, when in reality they’re at the mercy of the (largely default) claim behavior of third-party IdPs.

At worst, this can be a serious privacy and security concern, especially if composed with a key transparency or logging scheme: a user seeking to make public signatures may accidentally end up permanently disclosing personal details (or account security posture information).

How does Sigstore fare, by contrast? It too exposes some OIDC claim values, but in a much more controlled manner: claims are filtered through per-IdP allowlists before being embedded in issued certificates. As a result, neither the entire claim set nor any surprise claim additions by an IdP ultimately surface in Sigstore-issued certificates. Critically, the identity token itself is never shared beyond Sigstore’s services.

IdP behavior and expectations

Key binding and compatibility

As mentioned above: OpenPubKey essentially performs a hack to “trick” an identity provider into binding an identity credential to a user-controlled public key.

That “trick” is to stuff a public key identifier along with some other fields⁴ into a pre-existing user-controllable claim. The whitepaper mentions using the nonce claim for this purpose, but the reference implementation appears to use aud instead.

This discrepancy isn’t particularly noteworthy, except for what it implies: that the OpenPubKey developers discovered that many OIDC IdPs don’t actually support a user-controlled nonce at all, frequently because identity token retrieval is done through a dedicated REST API endpoint rather than a full OAuth2 flow.

More generally, however:

1. Support for these kinds of claim shenanigans is a mixed bag, and again falls into the gray space of “might be arbitrarily changed by an IdP tomorrow, breaking signature generation for everyone.”
2. The aud claim, in particular, is not always exposed as a fully user-controlled claim by many IdPs: it may instead be subject to an allowlist of audiences known to the OIDC provider, may have length or format restrictions, or may not even be a string (OIDC does not require this, and some IdPs prefer to encode it as an array of audiences).

Combined, these make for a somewhat shaky basis for a distributed verification system. Sigstore sidesteps these issues again through its use of trusted services: discrepancies between claim formats are handled by Fulcio’s use of Dex, and Sigstore currently only requires aud="sigstore" in terms of deoting the public good instance as the intended audience.

Expiration

I don’t have much to say about this, except for highlighting this section as probably a bad idea:

To enable verifying parties to enforce expiration if they so wish while avoiding the bad user experience of forcing the user to run the Authorization Code Flow every time the ID Token in their PK Token expires, we use the following alternative expiration mechanism. We do not expire a PK Token when the underlying ID Token expires according to its exp claim. Instead verifiers wishing to enforce expiration inspect the iat (issued at) claim, which specifies when the OP issued the ID Token, and reject the PK Token if it is older than two weeks. That is, if expiration is enforced, a PK Token expires two weeks after it is issued. (Page 9, S. 3.5.2)

Implicit expiration times like this are pretty dangerous, especially when describes in “may” language. See below for further thoughts on how this is likely to interact poorly with JWKS rotation practices, as well as difficult to reconcile with revocation.

Long-term signatures

I’ve put this section last because it isn’t fully clear to me that long-term signatures are an actual goal of the OpenPubKey design⁵. As such, the notes and concerns here may not be applicable. However, because the preprint mentions an “archival log” for handling verifications of older signatures, I felt that it’s appropriate to include my thoughts as well⁶.

One of the building blocks for OIDC is the JSON Web Key Set (JWKS) for a given service; this set is (typically) found through a .well-known discovery procedure and is comprised of individual JSON Web Key-formatted public keys.

JWKS allows OIDC to sidestep the hard problem of timely, reliable key rotation: if an IdP discovers that their signing material is compromised, they can remove the corresponding public key from their JWKS and effectively revoke it for all current users⁷.

Notably, JSON Web Keys do not carry their own expiries, revealing the other edge of the sword: an IdP may choose to rotate their keys at any point, and are not required to adhere to any particular schedule or sensible cadence.

Given that an OIDC IdP can choose to rotate their keys at any time and thereby render the current “batch” of OpenPubKey-bound tokens invalid, how does OpenPubKey ensure that signatures remain verifiable? This is what the preprint says:

The main challenge facing archival verifiers is that the OP public key necessary to verify the PK Token may not longer be available at the OP’s JWKS endpoint. Most OP’s (sic) rotate their signing keys out of JWKS endpoint between every two weeks to a four times a year. To ensure the verifier has the OP’s public key for the PK Token sent with the OSMs, the verifier must create and maintain an archival log of OP public keys. This archival log must cover the time period that verifier wishes to verify PK Tokens over. The verifier builds this archival log by regularly downloading the public keys from the OP’s JWKS endpoint. The verifier not only records the public keys but also the time at which the public keys were downloaded. (Page 9, S. 3.5.3)

This scheme is concerning for three reasons:

1. The “archival log” in question is underspecified. If you squint at it, it sounds a bit like a transparency log⁸, but inclusion enforcement and verifiable lookups are not mentioned directly⁹. In other words: if the archival log is itself a third-party service, then consumers of that service must effectively trust it even more than they would a verifiable transparency log (since the archival log can lie about its contents at any point).
2. (Seemingly) no concern is given to revocation: neither OIDC nor OIDC discovery provide a mechanism for clarifying that a key that has been rotated out of the JWKS has been rotated for reasons of compromise; it’s unclear how (if?) the archive log maintainer is intended to detect that case and express it within the log.
3. Completeness is not addressed: a user maintaining an archival log will presumably need to periodically poll their salient IdPs for new JWKS. Given that an IdP is perfectly within its rights to rotate its JWKS members at any time without warning, it’s unclear how an archival log maintainer can be confident that their log contains all of the keys needed to verify a given IdP’s identity tokens.

Of these concerns, (2) is (in my opinion) the most serious: OIDC itself contains no real mechanisms for long-term revocation, because its underlying key establishment mechanism is intended to avoid those problems in the first place. By reintroducing long-term public key handling in the form of an (unverifiable?) archival log, OpenPubKey effectively subverts OIDC’s primary mechanism for handling key material compromise.

Summary

Again, I’d like to emphasize that I think the idea behind OpenPubKey is extremely interesting and fundamentally valuable¹⁰. Moreover, I believe it deserves further attention (and development).

At the same time, it’s my current opinion that the current design of OpenPubKey does not inspire confidence, particularly with respect to long-term verification: there appear to be a lot of hacks that assume that arbitrary third-party OIDC IdPs are relatively static, predictable, and permissive of claim shenanigans, hacks that OIDC IdPs themselves are unlikely to be willing to ossify into guarantees (or, more precisely, cannot ossify without compromising their security practices).

By and large, GitHub Actions continues to delight me and grow new features that I appreciate: reusable workflows, OpenID connect, job summaries, integrations into GitHub Mobile, and so forth.

At the same time, GitHub Actions is a regular source of profound frustration and time loss² in my development processes. This post lists some of those frustrations, and how I think GitHub could selfishly³ improve on them (or even fix them outright)⁴.

Debugging like I’m 15 again

Here’s a pretty typical session of me trying to set up a release workflow on GitHub Actions:

In this particular case, it took me 4 separate commits (and 4 failed releases) to debug the various small errors I made: not using ${{ ... }}⁵ where I needed to, forgetting a needs: relationship, &c.

Here’s another (this time of a PR-creating workflow), from a few weeks later:

I am not the world’s most incredible programmer; like many (most?), I program intuitively and follow the error messages until they stop happening.

GitHub Actions is not responsible for catching every possible error I could make, and ensuring that every workflow I write will run successfully on the first try.

At the same time, the current debugging cycle in GitHub Actions is ridiculous: even the smallest change on the most trivial workflow is a 30+ second process of tabbing out of my development environment (context switch #1), digging through my browser for the right tab (context switch #2), clicking through the infernal nest of actions summaries, statuses, &c. (context switch #3), and impatiently refreshing a buffered console log to figure out which error I need to fix next (context switch #4). Rinse and repeat.

Fixing this

• Give us an interactive debugging shell, or (at least) let us re-run workflows with small changes without having to go through a git add; git commit; git push cycle⁶.

• Give us a repository setting to reject commits with obviously invalid workflows (things like syntax that can’t possibly work, or references to jobs/steps that don’t exist). It’s infuriating when I git push a workflow that silently fails because of invalid YAML; especially when I then merge that workflow’s branch under the mistaken impression that the workflow is passing, rather than not running at all.

Security woes

Speaking from experience: it’s shockingly easy to wreck yourself with GitHub Actions. Way easier than it should be.

Here is just a small handful of the ways in which I have personally written potentially vulnerable workflows over the past few years:

1. Using the ${{ ... }} expansion syntax in a shell or other context where a (potentially malicious) user controls the expansion’s contents. The following, for example, would allow a user to inject code that could then exfiltrate $MY_IMPORTANT_SECRET:

   1
   2
   3
   4
   5
   - name: do something serious
     run: |
      something-serious "${{ inputs.frob }}"
     env:
       MY_IMPORTANT_SECRET: ${{ secrets.MY_IMPORTANT_SECRET }}
   

   Some among you will observe that a ✨good✨ programmer would simply know not to do this, and that a bad programmer would eventually learn their (painful) lesson. This might be an acceptable position for a niche piece of software to hold; it is not an acceptable position for the CI/CD platform that, to a first approximation, hosts the entire open source ecosystem.

2. Using pull_request_target. As far as I can tell, it’s practically impossible to use this event safely in a non-trivial workflow⁷.

   This event appears to exist for an extremely narrow intended use case, i.e. labeling or commenting on PRs that come from forks. I don’t understand why GitHub Actions chooses to expose such a (relatively) simple operation through as massive of a foot-gun as pull_request_target.

3. Over-scoping my workflow and job-level permissions.

   The default access set for Actions’ ordinary GITHUB_TOKEN is very permissive: the only thing it doesn’t provide access to are the workflow’s OpenID Connect token.

   This consistently bites me in two different ways:

   1. I consistently forget to down-scope the default token, especially when working with repositories under my personal account (rather than under an org, where the default scope can be reduced across all repositories).

   2. I consistently over-scope my tokens because I don’t know exactly how much access my workflow will need.

      This is further complicated by the messy ways in which GitHub’s permission model gets shoehorned into a single permissions dimension of read/write/none: why does id-token: write grant me the ability to read the workflow’s OpenID Connect token? Why do some GET operations on security advisories require write, while others only require read?

There are also a few things that I haven’t done⁸, but are scary enough that I think they’re worth mentioning.

For example, can you see what’s wrong with this workflow step?

1
2
steps:
  - uses: actions/checkout@c7d749a2d57b4b375d1ebcd17cfbfb60c676f18e

Despite all appearances, SHA ref c7d749a2d57b4b375d1ebcd17cfbfb60c676f18e is not a commit on the actions/checkout repository! It’s actually a commit on a fork in actions/checkout’s network which, thanks to GitHub’s use of alternates, appears to belong to the parent repository.

Chainguard has an excellent post on this⁹, but to summarize:

1. SHA references from forks are visually indistinguishable from SHA references in the intended target repository. The only way to tell the two apart is to manually inspect each reference and confirm that it appears on the expected repository, and not one of its forks.
2. GitHub’s own REST API makes no distinction between SHA references in a repository graph — /repos/{user}/{repo}/commits/{ref} returns a JSON response that only references {user}/{repo}, even if {ref} is only on a fork.
3. Because GitHub fails to distinguish between fork and non-fork SHA references, forks can bypass security settings on GitHub Actions that would otherwise restrict actions to only “trusted” sources (such as GitHub themselves or the repository’s own organization).

GitHub’s response to this (so far) has been to add a little bit of additional language to their documentation, rather than to forbid misleading SHA references outright.

Fixing this

• Give us push-time rejection of obviously insecure workflows. In other words: let us toggle¹⁰ a “paranoid workflow security” mode that, when enabled, causes git push to fail with an explanation of what I’m doing wrong. Essentially the same thing as the debugging request above, but for security!

• Give us runtime checks on our workflows, analogous to runtime instrumentation like AddressSanitizer in the world of compiled languages. There are so many things that could be turned into hard failures for security wins without breaking 99.9% of legitimate users, like failing any attempt to use actions/checkout on a pull_request_target with a ref that isn’t from the targeted repository.

• Maybe just deprecate and remove pull_request_target entirely. GitHub’s own Security Lab has been aware of how dangerous this event is for years; maybe it’s time to get rid of it entirely.

• Allow us to set a more restrictive default token scope on our personal repositories, similar to how organizations and enterprises can restrict their default GITHUB_TOKEN scopes across all repositories at once.

• By default, reject any SHA-pinned action for which the SHA only appears on a fork and not the referenced repository. It’s hard to imagine a legitimate reason to ever need to do this!

Real types would be nice

When writing a custom GitHub Action, you can specify the actions inputs using a mapping under the inputs: key. For example, the following defines a frobulation-level input with a description (used for tooltips in many IDEs) and a default value:

1
2
3
4
inputs:
  frobulation-level:
    description: "the level to frobulate at"
    default: "1"

Notably, this syntax does not allow for type enforcement; the following does not work:

1
2
3
4
5
6
7
inputs:
  frobulation-level:
    description: "the level to frobulate to"
    default: 1
    # NOTE: this SHOULD cause a workflow failure if the input
    # isn't a valid number, but doesn't
    type: number

This absence is strange, but what makes it bizarre is that GitHub is inconsistent about where types can appear in actions and workflows:

• workflow_call supports type with boolean, number, or string
• workflow_dispatch supports type with boolean, choice, number, or string
• Action inputs: no types at all

Unfortunately, this is only the first level: even inputs that do support typing doesn’t support compounded data structures, like lists or objects. For example, neither of the following works:

1
2
3
4
5
6
7
8
- uses: example/example
  with:
    # INVALID: can't use arrays as inputs
    paths: [foo, bar, baz]
    # INVALID: can't use objects as inputs
    headers:
      foo: bar
      baz: quux

…which means that action writers end up requiring users to do silly things like these:

1
2
3
4
5
6
7
- uses: example/example
  with:
    # SILLY: action does ad-hoc CSV-ish parsing
    paths: foo,bar,baz
    # SILLY: action forcefully flattens a natural hierarchy
    header-foo: bar
    header-baz: quux

This is bad for maintainability, and bad for security: maintainability because actions must carefully manage a single flat namespace of inputs (with no types!), and security because both action writer and workflow writer are forced into ad-hoc, unspecified languages for complex inputs.

Fixing this

• Let action and workflow writers use type: everywhere, and let us use choice everywhere — not just in workflow_dispatch!

• Give us stricter type-checking. Where action and workflow types can be inferred statically, detect errors and reject incorrectly typed workflow changes at push time, rather than waiting for the workflow to inevitably fail.

• Give us type: object and type: array types. These won’t be perfect to start with (thanks to potentially heterogeneous interior types), but they’ll be a significant improvement over the status quo. Implementation-wise, forward these as JSON-serialized strings or something similar¹¹ where appropriate (such as in auto-created INPUT_{WHATEVER} environment variables).

(More) official actions would be nice

The third-party ecosystem on GitHub Actions is great: there are a lot of high-quality, easy-to-use actions being maintained by open source contributors. I maintain a handful of them!

Beneath the surface of these excellent third-party actions is a substrate of official, GitHub-maintained actions. These actions primarily address three classes of fundamental CI/CD activities:

1. Core git operations: actions/checkout
2. Core GitHub operations and repository housekeeping: actions/{upload,download}-artifact, actions/cache, actions/stale
3. General (but essential) configuration: actions/setup-python, actions/setup-node

These classes are somewhat distinct from “higher-level” workflows (like the kind I write): because of their centrality and universal demand, they benefit from singular, high-quality, officially maintained implementations.

And so, the question: why are there so few of them?

Here is just a smattering of the official actions that don’t exist:

1. Programmatically adding a pull request to a merge queue. GitHub has the machinery to support this: gh pr merge already exists. It just isn’t exposed as an action; users are (presumably) expected to piece it together themselves.

Even worse, there are actions that did exist but were deprecated (generally for unclear reasons¹²):

1. actions/create-release: unmaintained as of March 2021. Users encouraged to switch to various community maintained workflows, most notably¹³ softprops/action-gh-release.
2. actions/upload-release-asset: marked as unmaintained at the same time as actions/create-release.
3. actions/setup-ruby: unmaintained as of February 2021. Users encouraged to switch to ruby/setup-ruby.

I’m sympathetic to the individual maintainers here and, in each case, the transition to a “recommended” third-party action was relatively painless.

Still, the overall impression given here is unmistakable: that GitHub does not see official actions for its own platform features (or key ecosystem users, like Ruby) as priorities¹⁴, and would rather have the community develop and choose unofficial favorites. This is not unreasonable on a strategic level (it induces third-party development in their ecosystem), but has a deleterious effect on trust in the platform. I’d like to be able to write workflows and know that they’ll run (with minimal changes) 5 years from now, and not worry that GitHub has abandoned core pieces underneath me!

Apart from imparting a general feeling of shabbiness, this compounds with GitHub Action’s poor security story (per above): not providing official high-quality actions for their own API surfaces means that users will continue to make exploitable security mistakes in their workflows. Nobody wins¹⁵.

Fixing this

• Give us more official actions. As a very rough rule of thumb: if a thing directly ties different pieces of GitHub infrastructure together and currently needs to be done manually (with REST API calls, gh invocations, or whatever else), it probably deserves a full official action!

• Give us more pseudo-official actions. Work with the biggest third-party actions¹⁶ to form a community-actions (or whatever) org, with the expectation that actions homed under that org have been reviewed (at some point) by GitHub, are forced to adhere to best practices for repository security, receive semantically versioned updates, &c &c.

Wrap-up

This is a long and meandering post, and many parts are in conflict: security and stability (in the form of more official actions that break less often), for example, are in eternal conflict with each other.

I’m just one user, and I don’t expect my interests or frustrations to be overriding ones. Still, I hope that the problems (and potential fixes) above aren’t unique to me, and that there are engineers at GitHub who (again, selfishly!) share these concerns and would like to see them fixed.

Addendum (2023-09-26)

I got quite a few emails and private messages over this post with similar (excellent!) points and observations. This is my attempt to collect them:

• Several people recommended that I try nektos/act for local GitHub Actions emulation. Reader, let me tell you: I have! I’ve been using it for a few years, and it’s a nice tool for rapidly iterating on and debugging simple workflows. In my experience, however, it’s a lossy tool: I regularly run into situations where act’s images aren’t quite like GitHub’s, where events aren’t exactly the same as they appear on GitHub, &c. This isn’t the fault of the act maintainers: they’re a third-party trying to follow a largely proprietary ecosystem that has its own development pace. As a third-party tool that can’t be expected to obtain perfect compatibility, I didn’t want this post to attract unnecessary complaints (or misunderstandings) their way,

• Several people also recommended that I try rhysd/actionlint for local linting (including security linting) of GitHub Actions workflows. Like act, I have! I actually use actionlint much more regularly than I do act, both for professional development and my personal projects/contributions. Unfortunately, it has its own deficiencies (for my purposes, the biggest one is its inability to lint actions instead of just workflows).

• I was also recommended the GitHub Actions extension for VS Code, which provides additional in-editor linting (built on top of public JSON schemas for GitHub Actions) as well as integrations with a specific repository’s workflows (showing runs in progress, completing variables, &c). I’ve been using this extension for a few months, and it’s a significant improvement over writing workflows blindly. Still, I find that many of my workflows need a decent amount of add-commit-push debugging, due to things that the extension can’t detect (typos in secrets, typos in dynamically generated output names, &c).

Additionally, Julian Dunn, head of project management for GitHub Actions, reached out to me and pointed out some important features in the works, as well as some clarifications on their internal priorities:

• Interactive debugging is a part of GitHub Actions’ public roadmap.
• Similarly: GitHub Actions is actively looking to move away from tag- and SHA-based workflow pinning, and towards something more immutable. This is also on their public roadmap.
• Additionally, I was told that GitHub does see the health and quality of the official actions ecosystem as a priority, and that part of their (current) strategy for doing so is to keep the overall number of official actions small (to ensure that each receives sufficient maintenance and attention). This is a very understandable position, and I appreciate the candor in discussing it!

Quick context

Most Delta flights have in-flight Wi-Fi, including international flights, including the one I’m on right now.

Normally you’d have to pay for it (on international flights), which I refuse to do out of principle.

However, Delta currently has an agreement with T-Mobile, giving T-Mobile customers free in-flight Wi-Fi.

Very cool, thanks Delta and T-Mobile! Except for the fine print: while it’s a normal unrestricted¹ connection, it only works on mobile devices, not on laptops.

The justification for this is (probably) that Delta and T-Mobile can’t perform their little SMS verification flow from my laptop, but this is bull: iMessage can forward their SMS verification codes just fine, and besides there are other ways for the two to verify that I’m a T-Mobile subscriber (such as asking my for my T-Mobile credentials).

This wasn’t satisfying, so I came up with two quick workarounds. The first definitely works (it’s what I’m using to write this), and the latter probably works (it was what I was originally planning on using because it’s a little simpler, but requires you to not previously connect to inflight Wi-Fi on your phone).

Everything below assumes iOS and macOS, since it’s what I’m on right now, but nothing about either technique should require either OS.

Workaround 1: MAC spoofing

It’s deeply funny to me that this still works in 2023.

The steps here are pretty simple:

1. Connect to Delta’s inflight Wi-Fi on your phone and go through the T-Mobile SMS flow;
2. Confirm that you’re connected to the interwebz;

3. Open up your phone’s Wi-Fi settings and make a note of your Wi-FI interface’s MAC

   I have no idea where this information appears on Android. On iOS, it’s under the little “info” button for the Wi-Fi network you’re currently on.

4. Turn off your phone’s Wi-Fi

   We don’t want to confuse the inflight Wi-Fi more than necessary by having two devices with the same MAC appear simultaneously.

5. On macOS, make sure that any previous configuration for Delta’s inflight Wi-Fi has been deleted.

6. Force macOS to disassociate from any networks:

   1
    sudo /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -z
   

7. Set the MAC for your Wi-Fi interface to the MAC that your phone was using²:

   1
   2
    sudo ifconfig en0 ether AA:BB:CC:AA:BB:CC
    sudo ifconfig en0 lladdr AA:BB:CC:AA:BB:CC
   

   …where en0 is your laptop’s Wi-Fi interface and AA:BB:CC:AA:BB:CC is your phone’s Wi-Fi interface’s MAC.

   I’m not sure if both of these are required or not, but experimentally using both didn’t hurt things.

8. Connect to Delta’s inflight Wi-Fi again, and enjoy your internet:

   

Workaround 2: User agent spoofing

I’m including this one for good measure, since I’m pretty sure it would work and is slightly easier (when you do it right). It has a few constraints:

1. You can’t have previously connected to Delta’s inflight Wi-Fi with your phone, since we aren’t spoofing the MAC and T-Mobile only seems to allow one MAC per flight.
2. Your laptop needs to be able to receive SMS messages somehow (e.g. via iMessage with your provider’s support for SMS over Wi=Fi).

With this technique, you just change your browser’s user agent to something that Delta’s interstitial recognizes as a mobile phone to get it to show you the T-Mobile option.

For Safari, this can be done by enabling Developer Mode (Settings > Advanced > Show Develop menu in menu bar) and setting the user agent to an iPhone agent:

From there, navigate to Delta’s inflight interstitial (https://wifi.inflightinternet.com) and use the newly visible T-Mobile option, going through SMS verification as normal.

I’m pretty sure this will work, assuming you have Wi-Fi Calling enabled and have iMessage on your Mac linked to your phone.

Summary

Neither of these tricks is new; they’re both very old!

That being said, I was amused (and a little endeared) by the fact that I could use them to get online with inflight Wi-Fi in 2023 — I guess things on the device sniffing/doing-better-than-trusting-MAC front haven’t improved much over the last decade (or, equally likely, Delta and T-Mobile just don’t care about the tiny fraction of users who can figure this out).

On the off chance they do care about this and haven’t intentionally left this gap open for more motivated users, I welcome whatever trick they use to stop my from doing this. It’ll make for an interesting challenge on my next flight.

As the demo implies, shaq does just one thing: it listens to an audio source and sends the results to Shazam for fingerprinting. If a match is found, it prints it.

As usual, you can install it from PyPI using pip or pipx:

1
2
3
4
5
pip install shaq
# or:
pipx install shaq

shaq --help

Usage

By default shaq listens to the system microphone (helpfully supplied through portaudio) and writes its findings as plain text, but you can tell it to detect from an arbitrary input instead:

1
2
3
4
5
6
7
8
# shaq analyzes the first 10 seconds by default
shaq --input mystery.mp3

# analyze a longer segment
shaq --input mystery.mp3 --duration 15

# anything with an audio track that ffmpeg can handle can be an input
shaq --input another-mystery.mp4

…as well as to emit JSON:

1
shaq --listen --duration 5 --json | jq '[.track.title, .track.subtitle]'

produces:

1
2
3
4
[
  "Mendacity",
  "Max Roach"
]

Under the hood, shaq is a relatively thin wrapper around PyAudio, pydub, and shazamio.

For the moment, shaq only supports fixed durations. Once I get bored again, I plan to add:

• Support for “rolling” detections, i.e. “listen until you get a match”;
• Support for timeranges in file inputs, i.e., “15 seconds starting at 00:12:13”.

But why?

I am a compulsive scrobbler; I record almost everything I listen to, including phonograph records.

For records, I currently use Vinyl Scrobbler to scrobble records as I play them and it works very nicely (it even puts scrobbles in the future so that they end up mostly synchronized with the actual record!).

At the same time, it’s slightly more manual than I’d prefer: I’d like to be able to put a record on and have scrobbles come through automatically as each track starts. This is why I made shaq: my plan is to hook the (unused) headphones channel on my received into a Raspberry Pi, which will then run shaq either continuously or on a transition trigger (i.e., noise drop between tracks).

The Pi will then scrobble based on shaq’s results, satisfying my desire to be simultaneously too lazy to click a single button on my computer while also physically flipping big chunks of vinyl every 20 minutes.

I’ve summarized each below; if you’re a user or previous contributor to one, please get in touch with me! I’m also happy to consider non-users and entirely new maintainers, although I’d like to understand your potential interest and/or stake in the project if you aren’t already a user or contributor.

ruby-mpv

ruby-mpv is a Ruby library for controlling MPV processes through its JSON IPC protocol. I created this library a few years ago for a side project that has now been abandoned and memory holed, but the library itself is general purpose and already covers a large chunk of MPV’s IPC API.

qrencode.cr

qrencode.cr is a Crystal library that exposes high-level bindings for libqrencode.

x_do.cr

x_do.cr is a Crystal library that exposes high-level bindings for libxdo.

notify.cr

notify.cr is a Crystal library that exposes high-level bindings for the D-Bus Desktop Notifications service.

libbdiff

libbdiff is a linkable C library version of Colin Percival’s bsdiff. It hasn’t been maintained in nearly a decade.

ruby-inih

ruby-inih is a native (meaning written in C) Ruby library that exposes the inih INI parser via a Ruby API.

SimpleSession

SimpleSession is a bare-bones session manager for Sublime Text 3 and 4, intended to be a simpler and more sync-friendly version of Sublime Text’s built-in “workspace” support. Based on Package Control statistics, it probably has a few dozen users.

Others?

These were just the few that I could remember — there are almost certainly others. If you’ve contributed to a project of mine that isn’t listed above and you’d like to help maintain it, please don’t hesitate to get in touch as well!

Background

I have a friend who runs an Internet radio server; I do a (mostly) weekly show on it.

I’ve done Internet radio before (and ran my own personal server for a few years), and wasn’t particularly happy with the tooling or broadcasting flows: most flows were either extremely inflexible (fixed playlists) or brittle (local loopbacks that sink to a tool like butt, requiring error-prone fiddling with my local sound settings).

In contrast to both of those, I wanted something that could:

• Handle both a “primary” playlist and live microphone break-ins seamlessly, without requiring me to perform a hacky loopback with PulseAudio or ALSA;
• Integrate into a larger station management workflow, including automatically fetching (and streaming) the right playlist from my Navidrome instance;
• Be maintained as a program, rather than a collection of semi-reproducible pieces of global system state tied together with shell scripts.

These properties brought me to Liquidsoap¹.

Liquidsoap: a Swiss Army knife for Internet radio

Liquidsoap is, among other things, an entire programming language dedicated to describing and composing audio streams².

Conceptually, Liquidsoap turns inputs (a streaming playlist, live microphone input, periodic jingles and announcements, &c) into a stream generator, which is then sunk into outputs (the local audio output, an MP3 backup, an Icecast server, &c).

This is all done with strongly typed ML-ish³ scripts, like the following:

1
2
3
4
5
6
7
8
9
10
source = mksafe(playlist("radio.m3u", mode="normal"))

output.icecast(
  %mp3(bitrate=320, samplerate=44100, stereo=true),
  mount="/stream",
  host="server.example.com",
  port=8000,
  password="hunter2",
  description="my first radio stream",
  source)

The above is a very basic example: all it does it take an M3U-formatted playlist and stream it to an Icecast server.

One of Liquidsoap’s key niceties is infallibility: the language won’t let you define a stream generator that will fail to provide input when the sink needs it. That’s what the mksafe function in the example above does: it converts a fallible source (a playlist) into an infallible one by ensuring that radio silence is streamed if the underlying playlist is malformed or can’t be streamed in time.

We can demonstrate by trying to remove the mksafe and observing the typechecking error Liquidsoap gives us:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# this version may be pretty old;
# see https://www.liquidsoap.info/doc-dev/install.html
sudo apt install -y liquidsoap

liquidsoap 'source = playlist("radio.m3u", mode="normal")

output.icecast(
  %mp3(bitrate=320, samplerate=44100, stereo=true),
  mount="/stream",
  host="server.example.com",
  port=8000,
  password="hunter2",
  description="my first radio stream",
  source)'

At line 0, char 9-45:

Error 7: Invalid value:
That source is fallible

With these building blocks (fallible and infallible sources, stream generators, and outputs) we can begin to do more complex things, like defining crossfades between tracks:

1
2
3
source = playlist("radio.m3u", mode="normal")
source = crossfade(duration=2.0, source)
source = mksafe(source)

…or adding an additional input source (a microphone), and mixing it in with a fade in/out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# music source: playlist
music = mksafe(playlist("radio.m3u", mode="normal"))

# microphone source: amplify to 130% and strip blanks > 0.75 secs
mic = input.alsa(bufferize=false)
mic = amplify(1.3, mic)
mic = blank.strip(max_blank=0.75, mic)

def f(a, b)
  add(normalize=false, [fade.out(a), fade.in(b)])
end

# fade in and out of the mic/music sources
mix = fallback(
  track_sensitive=false,
  transition_length=0.75,
  transitions=[f, f],
  [mic, music])

Liquidsoap also has a mature callback system, allowing us to define behaviors that should occur e.g. whenever the stream’s metadata changes:

1
2
3
# scrobble the stream's metadata whenever it changes (i.e., when a
# new song begins)
music.on_metadata(fun(m) -> lastfm.submit(user="***", password="***", m))

These examples are just the tip of the iceberg; the Core API and Extra API documentation contains far more, including all kinds of neat signals processing and advanced filtering functionality that I haven’t needed (yet).

Typing it all together

Here is what my (admittedly hacky) Liquidsoap-defined Internet radio looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
#!/usr/bin/liquidsoap -v

log.stdout.set(true)

test = getenv("TEST") != ""
mount = if test then "/test" else "/stream" end
date = argv(default=time.string("%Y-%m-%d"), 1)

# Do your own thing here.
radio_password = process.read("kbs2 pass badradio.biz")

# Do your own thing here.
lastfm_username = "yossarian_flew"
lastfm_password = process.read("kbs2 pass last.fm")

# retrieve the playlist
playlist_name = date ^ ".m3u"
playlist_path = "/tmp/" ^ playlist_name

print("writing temp playlist to: " ^ playlist_path)

if process.test("ruby gimme-playlist " ^ date ^ " > " ^ playlist_path) then
  log.important("gimme-playlist: saved to " ^ playlist_path)

  def cleanup()
    file.remove(playlist_path)
  end

  on_shutdown(cleanup)
else
  print("gimme-playlist failed for " ^ date)
  exit(1)
end

music = mksafe(crossfade(duration=2.0, playlist(playlist_path, mode="normal")))

# TODO: Use lastfm.submit.full instead, once it works with crossfades.
# See: https://github.com/savonet/liquidsoap/issues/3172
# music = lastfm.submit.full(user=lastfm_username, password=lastfm_password, music)
music.on_metadata(fun(m) -> lastfm.submit(user=lastfm_username, password=lastfm_password, m))

# microphone source: amplify to 130% and strip blanks > 0.75 secs
mic = input.alsa(bufferize=false)
mic = amplify(1.3, mic)
mic = blank.strip(max_blank=0.75, mic)

def f(a, b)
  add(normalize=false, [fade.out(a), fade.in(b)])
end

# fade in and out of the mic/music sources
mix = fallback(
  track_sensitive=false,
  transition_length=0.75,
  transitions=[f, f],
  [mic, music])

output.alsa(bufferize=false, mix)

output.icecast(
  %mp3(bitrate=320, samplerate=44100, stereo=true),
  mount=mount,
  host="server.badradio.biz",
  port=8000,
  password=radio_password,
  url="https://yossarian.net/junk/badradio/" ^ date,
  description="the rummage bin",
  mix)

Starting the stream is as simple as:

1
2
3
4
5
# `radio.liq` is the source file
liquidsoap radio.liq -- "2023-06-26"

# stream to the test endpoint instead
TEST=1 liquidsoap radio.liq -- "2023-06-26"

This does a few things automatically for me:

• It handles both test and showtime streaming; I set TEST=1 while invoking the script to switch to a testing endpoint.
• It keeps my credentials out of the program: the radio’s definition uses kbs2 (my password manager) to retrieve the appropriate credentials as required.
• It automatically sets up the right playlist: I pass in the show’s date, and the radio’s definition shells out to a little helper (gimme-playlist) that talks to my Navidrome instance and generates an M3U-formatted playlist. It also ensures that the playlist is cleaned up automatically, via the on_shutdown hook.
• It makes source handling “intuitive”: I don’t have to configure anything ahead-of-time, and switching to my microphone is as simple as unmuting myself; the stream gracefully handles the transition with a fade.

All told, Liquidsoap has made a normally stressful and unappealing task into a genuinely fun one: it’s allowed me to spend less time thinking about ways in which my stream can break catastrophically, and more time actually planning my shows. It’s also given me plenty of ideas for future improvements, including:

• Further automating my pre- and post-show operations, like updating the show’s website.
• Attempting some kind of call-in and/or shoutout functionality, using the server’s chat.

Preword

I’ve been sitting on this post for a few months, in part because of travel and in part because its (intended) scope was beginning to reflect PGP’s own fractal complexity.

The version that I’m publishing now has been significantly pared down to remove extended digressions on how bad PGP’s packet format is, all the different ways in which a signature or certificate packet can be broken, incorrectly bound, &c.

I’ve removed those things because I think the results, as present, are sufficient evidence for the actual claims I’d like to make, namely:

1. That existing PGP signatures on PyPI serve no security purpose, and that all evidence points to nobody ever attempting to verify them;

2. Even advanced technical communities, as a whole, largely fail to reduce PGP’s complexity and unnecessary agility into a reasonable and tractable subset.

And, just in case it needs to be said:

1. This post isn’t intended to disparage PyPI: PyPI has done everything right, including purposely removing frontend support for PGP years ago.

2. This post isn’t intended to disparage individual packagers and maintainers still uploading signatures to PyPI. I suspect that much of the ongoing signature uploading is a result of long-forgotten automation and, even when it isn’t: developers cannot be blamed for their misuse of obtuse tools. Security tools, especially cryptographic ones, are only as good as their least-informed¹ and most distracted user.

Background

PyPI has supported PGP signatures in some form or another for a very long time².

To this date, PGP is still (minimally) supported: package uploaders can still sign for their package distributions and upload the resulting .asc to PyPI for inclusion in the index. The official uploading utility even supports invoking gpg directly via the --sign and --sign-with arguments!

To a novice Python programmer looking to publish their first package to PyPI, this might give the following impressions:

1. That PGP offers secure and modern cryptographic primtives;
2. That PyPI encourages users to upload PGP signatures or that doing so is best practice;
3. That others expect PGP signatures, and that package adoption is (in part) predicated on supplying PGP signatures.

The first two are just wrong:

1. PGP is an insecure and outdated ecosystem that hasn’t reflected cryptographic best practices in decades.

2. PyPI’s support is vestigial in nature: signatures are not shown as part of the web interface, and are only obliquely referenced in the PEP 503 and JSON APIs.

The third is harder to immediately refute: PyPI still hosts signatures, after all. Absent any other information, it’s entirely possible that companies and end users are quietly and diligently verifying whatever signatures are present, using trust sets, tracking revoked and expired keys, and so forth.

Thus, my goal with this blog post:

1. Determine how many signatures are on PyPI;
2. Correlate those signatures to their signing keys;
3. Analyze those signing keys for their practical value: their strength, liveness, &c.

Methodology

Relatively early in the process I decided not to collect every single signature on PyPI, for two main reasons:

1. Relevance: PyPI hosts many old package distributions, including distributions for Python 2.7 (and earlier!). Given that Python 2 has been EOL for over three years at this point, it didn’t feel relevant (or efficient) to retrieve large quantities of signatures that nobody is likely to ever try install the distributions for.

2. Fairness: both PGP and Python have a lot of history, much of which predates modern understandings around cryptographic best practices. Given that, it didn’t feel fair to analyze extremely old signatures, especially if doing so would bias the statistics away from newer users who are doing more responsible things.

Given these considerations, I decided to limit my analysis to only signatures uploaded to PyPI on or after 2020-03-27. I chose that date somewhat arbitrarily³ while also satisfying a few constraints:

• It’s well after the 2018 deployment of the new PyPI, which didn’t emphasize support for PGP signatures (while still retaining it). In other words: signatures uploaded in 2020 or later were either done by automation (implying some degree of sophistication) or were likely a conscious decision by a packager to continue signing with PGP.

• It’s very recent, and best practices around digital signatures have not changed substantially since 2020. In other words: a best-practices signature (and key) made in 2020 should look very similar to a best-practices signature (and key) made in 2023, and someone signing in 2020 would have no good excuses for not making reasonable choices.

Actually retrieving the signatures was a multi-step process. To start, I used PyPI’s BigQuery dataset to give me some basic metadata on every distribution file with an associated signature:

1
2
3
4
SELECT name, version, filename, python_version, blake2_256_digest
FROM `bigquery-public-data.pypi.distribution_metadata`
WHERE has_signature
AND upload_time > TIMESTAMP("2020-03-27 00:00:00")

This produced 52900 distributions uploaded since 2020-03-27 for which PyPI also had a signature (subtract 1 for the CSV header):

1
2
3
4
5
6
$ wc -l inputs/dists-with-signatures.csv
52901 inputs/dists-with-signatures.csv

$ head -2 inputs/dists-with-signatures.csv
name,version,filename,python_version,blake2_256_digest
pantsbuild.pants.testutil,1.30.0,pantsbuild.pants.testutil-1.30.0-py36.py37.py38-none-any.whl,py36.py37.py38,7ecbe47906ddbe8a2f1ee2505c2edb7f9313348d4925855e429be1d316660a00

From here, I needed to retrieve each release distribution’s detached signature, i.e. the adjacent .asc URL in PyPI’s object storage.

I initially did this with the “conveyor” service, which turns PEP 491 names into URLs like so:

1
https://files.pythonhosted.org/packages/source/{version}/{name[0]}/{name}/{dist}.asc

However, this was pretty lossy: for whatever reason⁴ my URLs were slightly off about 20% of the time, resulting in lots of missed signatures. I eventually realized that the BigQuery dataset also includes the Blake2 digest for each distribution, meaning that I could use the actual package URLs instead:

1
https://files.pythonhosted.org/packages/{digest[0:2]}/{digest[2:4]}/{digest[4:]}/{dist}.asc

…and this was perfectly reliable.

From here, I wanted to figure out (roughly) how many unique keys produced these ~50k signatures. I decided to use PGPy⁵ for that; excerpted from dists-by-keyid.py:

1
2
3
4
5
6
7
8
9
10
11
sig = pgpy.PGPSignature.from_blob(sig_resp.content)
try:
    # https://github.com/SecurityInnovation/PGPy/issues/433
    sig
    sig.signer
except AttributeError:
    print("barf: couldn't get signer, probably ancient", file=sys.stderr)
    _KEY_ID_MAP["<invalid signer>"].append(rec)
    continue

_KEY_ID_MAP[sig.signer].append(rec)

This left me with a big map of PGP key IDs⁶ to a list of distributions signed by them, including 26 distributions whose signatures PGPy couldn’t parse:

This is a tiny failure (26 distributions out of 52900, or roughly 0.5%), but it sets the tone for the rest of the post.

Apart from these 26 failures, the remaining 52874 signatures were produced from 1067 “unique”⁷ PGP keys.

Results

At this point, I had 1067 unique key IDs, each of which needed to be retrieved from a keyserver.

My expectation was that this wouldn’t be a significant challenge, despite the widely publicized implosion of the SKS keyserver network back in 2018: there are still a few major keyservers running, and package authors pushing to PyPI should have the presence of mind to upload their keys. Right?

Pictured: your author immediately before trying to retrieve PGP keys in 2023.

Wrong. Of the 1067 keys IDs collected through signatures on PyPI, a full 308 (or roughly 29%) had no publicly discoverable key on the major remaining keyservers. In other words: roughly 1/3rd of all signatures added to PyPI since 2020 are bound to keys that aren’t discoverable by the PGP ecosystem’s own tooling. They might exist, hidden on personal domains and documentation pages, but, for all intents and purposes, these 29% of keys are useless⁸.

So, our first graphic of the post: discoverable keys versus undiscoverable ones:

Pictured: a very normal and healthy signing ecosystem.

That left 759 discovered keys to actually audit. To keep things simple⁹, I limited my analysis to just the following considerations:

• Does the key’s certificate have a binding signature¹⁰?

• What algorithm does the key use?

  • Additionally, if it’s a subkey, what algorithm does the primary key use?
  • For RSA keys, what parameters does the key use?

If that seems like a limited analysis, it’s because it is: there are too many ways to produce a weirdly shaped PGP certificate and/or key packet sequence, and the existing tooling (things like pgpdump and pgp --with-colons) weren’t up to the task.

Instead, I wrote a little tool (pgpkeydump) to give me machine-readable dumps of PGP keys¹¹, and then wrapped it in a bulk auditing script that does some basic statistics on the results.

To summarize the results:

• Of the 759 discovered keys, 298 (39%) had no binding signature at their specified creation time. In other words: these keys’ certificates came with no verifiable proof for an associated identity, expiry, or any of the other basic metadata conceptually associated with a PGP key, including its intended purpose.
• 375 (49%) had no binding signature at the time of the audit (2023-05-19), meaning that any binding signature that was present had already expired. In other words: half of all keys used to sign on PyPI since 2020 are already expired. This strongly suggests that nobody is attempting to verify signatures from PyPI on any meaningful scale.

Then, on the algorithm and parameter sides¹²:

Primary keys:

“Effective”¹³ keys:

Or again, as pretty charts:

First, the “good” parts:

1. While normally a bad choice, RSA is literally the best you can do in terms of standard¹⁴ asymmetric signing algorithms in PGP. Over two thirds of keys used to sign on PyPI are using it, and they’re using reasonable¹⁵ key sizes (4096 and 3072).

Then, the meh:

1. A sizeable minority (20% of effective keys, and 17% of primary keys) are RSA-2048. NIST considers RSA-2048 to be equivalent to roughly 112 bits of security¹⁶, and does not recommend its use on data that’s expected to have a security life of 15 years…starting in 2015. That means that PyPI-hosted signatures against RSA-2048 keys have roughly 7 years of “shelf life” in them. Version turnover in packaging ecosystems has accelerated over the last decade; let’s hope that applies here too!

2. Some enterprising people are on the “bleeding edge”: they’re using EdDSA and a few different ECDSA curves. It’s hard to say whether this is good or bad: it’s good in the sense that these are almost certainly better than anything offered by strictly RFC 4880 PGP implementations, but pointless in the sense that support for verifying these signatures is limited¹⁷ to just a few clients. It’s also probably pointlessly slow (for P-521 and brainpoolP512r1 in particular).

And finally, the insane:

• Roughly 5% of all keys used to sign for packages on PyPI are DSA. The majority of those are DSA-1024, which is roughly equivalent in strength to RSA-1024. DSA of any size is already very bad, and DSA-1024 is well outside of any acceptable safety margin for signatures in 2023, much less 2020 or even 2010.

• RSA-4064 and RSA-4032. I have no idea why anyone would do this¹⁸. Maybe some misguided attempt to calculate a precise security margin, or a misreading of someone else’s recommendations?

• One of the RSA-2048 keys has a public exponent of 41, rather than 65537 (which every other RSA key in the dataset uses). Again, I have no idea why anyone would do this: it’s pointlessly slower and opens up padding concerns that e = 65537 is resilient against.

Takeaways

To summarize: of just the PGP signatures uploaded to PyPI in the last three years:

• A small handful (26) are so malformed or old that they can’t be correlated to a key ID;
• Of those that can be correlated to a key ID, nearly a third cannot be discovered through the PGP ecosystem’s intended key discovery mechanism;

• Of the remaining discoverable keys:

  • Nearly half (49%) have no active binding signature when retrieved from a keyserver, giving them indefinite (at the absolute best) identity properties.

  • A sizeable minority (20%) are using weak RSA keys with less than a decade before NIST considers them insecure.

  • A smaller but still appreciable minority (5%) are using DSA-1024 keys, which have been considered insecure for well over a decade.

By all rights, these numbers represent the best possible case for PGP signatures on PyPI. Expanding the audit to 2015 or even earlier would likely reveal far worse practices.

In one sense, none of this is a problem: the breadth and depth of issues here suggests that nobody (thankfully!) is actually relying on these signatures, and the continued presence of new signatures on PyPI is primarily a vestige of forgotten automation and outdated tutorials.

On the other hand, these results present a strong case against attempting to “rehabilitate” PGP signatures for PyPI, or any other packaging ecosystem: all evidence points to end users (i.e., signers) being unable¹⁹ to distinguish between the “good” and “bad” parts of PGP, much less use them at all (e.g. keyservers).

So, for final conclusions:

• Given how broken the PGP signatures and keys present on PyPI are, it’s unlikely that anybody is currently doing wide-scale verification against them.
• If anybody is (and I’d be interested to hear if you are!), then it’s almost certainly inadvisable: “verifying” these signatures is, on average, likely to provide a false degree of confidence in their value.

As with previous posts, I’ve tried to make my steps and data reproducible, and have checked them all into this repo. I welcome any discoveries of mistakes I’ve made, as well as any attempts to improve the overall detail or fidelity of the results!