ENOSUCHBLOG

Programming, philosophy, pedaling.

• Home
• Tags
• Series
• Favorites
• Archive
• Main Site

PGP signatures on PyPI: worse than useless

May 21, 2023     Tags: cryptography, devblog, programming, python, rant    

TL;DR: A large number of PGP signatures on PyPI can’t be correlated to any well-known PGP key and, of the signatures that can be correlated, many are generated from weak keys or malformed certificates. The results suggest widespread misuse of GPG and other PGP implementations by Python packagers, with said misuse being encouraged by the PGP ecosystem’s poor defaults, opaque and user-hostile interfaces, and outright dangerous recommendations.

Preword

I’ve been sitting on this post for a few months, in part because of travel and in part because its (intended) scope was beginning to reflect PGP’s own fractal complexity.

The version that I’m publishing now has been significantly pared down to remove extended digressions on how bad PGP’s packet format is, all the different ways in which a signature or certificate packet can be broken, incorrectly bound, &c.

I’ve removed those things because I think the results, as present, are sufficient evidence for the actual claims I’d like to make, namely:

1. That existing PGP signatures on PyPI serve no security purpose, and that all evidence points to nobody ever attempting to verify them;

2. Even advanced technical communities, as a whole, largely fail to reduce PGP’s complexity and unnecessary agility into a reasonable and tractable subset.

And, just in case it needs to be said:

1. This post isn’t intended to disparage PyPI: PyPI has done everything right, including purposely removing frontend support for PGP years ago.

2. This post isn’t intended to disparage individual packagers and maintainers still uploading signatures to PyPI. I suspect that much of the ongoing signature uploading is a result of long-forgotten automation and, even when it isn’t: developers cannot be blamed for their misuse of obtuse tools. Security tools, especially cryptographic ones, are only as good as their least-informed¹ and most distracted user.

Background

PyPI has supported PGP signatures in some form or another for a very long time².

To this date, PGP is still (minimally) supported: package uploaders can still sign for their package distributions and upload the resulting .asc to PyPI for inclusion in the index. The official uploading utility even supports invoking gpg directly via the --sign and --sign-with arguments!

To a novice Python programmer looking to publish their first package to PyPI, this might give the following impressions:

1. That PGP offers secure and modern cryptographic primtives;
2. That PyPI encourages users to upload PGP signatures or that doing so is best practice;
3. That others expect PGP signatures, and that package adoption is (in part) predicated on supplying PGP signatures.

The first two are just wrong:

1. PGP is an insecure and outdated ecosystem that hasn’t reflected cryptographic best practices in decades.

2. PyPI’s support is vestigial in nature: signatures are not shown as part of the web interface, and are only obliquely referenced in the PEP 503 and JSON APIs.

The third is harder to immediately refute: PyPI still hosts signatures, after all. Absent any other information, it’s entirely possible that companies and end users are quietly and diligently verifying whatever signatures are present, using trust sets, tracking revoked and expired keys, and so forth.

Thus, my goal with this blog post:

1. Determine how many signatures are on PyPI;
2. Correlate those signatures to their signing keys;
3. Analyze those signing keys for their practical value: their strength, liveness, &c.

Methodology

Relatively early in the process I decided not to collect every single signature on PyPI, for two main reasons:

1. Relevance: PyPI hosts many old package distributions, including distributions for Python 2.7 (and earlier!). Given that Python 2 has been EOL for over three years at this point, it didn’t feel relevant (or efficient) to retrieve large quantities of signatures that nobody is likely to ever try install the distributions for.

2. Fairness: both PGP and Python have a lot of history, much of which predates modern understandings around cryptographic best practices. Given that, it didn’t feel fair to analyze extremely old signatures, especially if doing so would bias the statistics away from newer users who are doing more responsible things.

Given these considerations, I decided to limit my analysis to only signatures uploaded to PyPI on or after 2020-03-27. I chose that date somewhat arbitrarily³ while also satisfying a few constraints:

• It’s well after the 2018 deployment of the new PyPI, which didn’t emphasize support for PGP signatures (while still retaining it). In other words: signatures uploaded in 2020 or later were either done by automation (implying some degree of sophistication) or were likely a conscious decision by a packager to continue signing with PGP.

• It’s very recent, and best practices around digital signatures have not changed substantially since 2020. In other words: a best-practices signature (and key) made in 2020 should look very similar to a best-practices signature (and key) made in 2023, and someone signing in 2020 would have no good excuses for not making reasonable choices.

Actually retrieving the signatures was a multi-step process. To start, I used PyPI’s BigQuery dataset to give me some basic metadata on every distribution file with an associated signature:

1
2
3
4
SELECT name, version, filename, python_version, blake2_256_digest
FROM `bigquery-public-data.pypi.distribution_metadata`
WHERE has_signature
AND upload_time > TIMESTAMP("2020-03-27 00:00:00")

This produced 52900 distributions uploaded since 2020-03-27 for which PyPI also had a signature (subtract 1 for the CSV header):

1
2
3
4
5
6
$ wc -l inputs/dists-with-signatures.csv
52901 inputs/dists-with-signatures.csv

$ head -2 inputs/dists-with-signatures.csv
name,version,filename,python_version,blake2_256_digest
pantsbuild.pants.testutil,1.30.0,pantsbuild.pants.testutil-1.30.0-py36.py37.py38-none-any.whl,py36.py37.py38,7ecbe47906ddbe8a2f1ee2505c2edb7f9313348d4925855e429be1d316660a00

From here, I needed to retrieve each release distribution’s detached signature, i.e. the adjacent .asc URL in PyPI’s object storage.

I initially did this with the “conveyor” service, which turns PEP 491 names into URLs like so:

1
https://files.pythonhosted.org/packages/source/{version}/{name[0]}/{name}/{dist}.asc

However, this was pretty lossy: for whatever reason⁴ my URLs were slightly off about 20% of the time, resulting in lots of missed signatures. I eventually realized that the BigQuery dataset also includes the Blake2 digest for each distribution, meaning that I could use the actual package URLs instead:

1
https://files.pythonhosted.org/packages/{digest[0:2]}/{digest[2:4]}/{digest[4:]}/{dist}.asc

…and this was perfectly reliable.

From here, I wanted to figure out (roughly) how many unique keys produced these ~50k signatures. I decided to use PGPy⁵ for that; excerpted from dists-by-keyid.py:

1
2
3
4
5
6
7
8
9
10
11
sig = pgpy.PGPSignature.from_blob(sig_resp.content)
try:
    # https://github.com/SecurityInnovation/PGPy/issues/433
    sig
    sig.signer
except AttributeError:
    print("barf: couldn't get signer, probably ancient", file=sys.stderr)
    _KEY_ID_MAP["<invalid signer>"].append(rec)
    continue

_KEY_ID_MAP[sig.signer].append(rec)

This left me with a big map of PGP key IDs⁶ to a list of distributions signed by them, including 26 distributions whose signatures PGPy couldn’t parse:

This is a tiny failure (26 distributions out of 52900, or roughly 0.5%), but it sets the tone for the rest of the post.

Apart from these 26 failures, the remaining 52874 signatures were produced from 1067 “unique”⁷ PGP keys.

Results

At this point, I had 1067 unique key IDs, each of which needed to be retrieved from a keyserver.

My expectation was that this wouldn’t be a significant challenge, despite the widely publicized implosion of the SKS keyserver network back in 2018: there are still a few major keyservers running, and package authors pushing to PyPI should have the presence of mind to upload their keys. Right?

Pictured: your author immediately before trying to retrieve PGP keys in 2023.

Wrong. Of the 1067 keys IDs collected through signatures on PyPI, a full 308 (or roughly 29%) had no publicly discoverable key on the major remaining keyservers. In other words: roughly 1/3rd of all signatures added to PyPI since 2020 are bound to keys that aren’t discoverable by the PGP ecosystem’s own tooling. They might exist, hidden on personal domains and documentation pages, but, for all intents and purposes, these 29% of keys are useless⁸.

So, our first graphic of the post: discoverable keys versus undiscoverable ones:

Pictured: a very normal and healthy signing ecosystem.

That left 759 discovered keys to actually audit. To keep things simple⁹, I limited my analysis to just the following considerations:

• Does the key’s certificate have a binding signature¹⁰?

• What algorithm does the key use?

  • Additionally, if it’s a subkey, what algorithm does the primary key use?
  • For RSA keys, what parameters does the key use?

If that seems like a limited analysis, it’s because it is: there are too many ways to produce a weirdly shaped PGP certificate and/or key packet sequence, and the existing tooling (things like pgpdump and pgp --with-colons) weren’t up to the task.

Instead, I wrote a little tool (pgpkeydump) to give me machine-readable dumps of PGP keys¹¹, and then wrapped it in a bulk auditing script that does some basic statistics on the results.

To summarize the results:

• Of the 759 discovered keys, 298 (39%) had no binding signature at their specified creation time. In other words: these keys’ certificates came with no verifiable proof for an associated identity, expiry, or any of the other basic metadata conceptually associated with a PGP key, including its intended purpose.
• 375 (49%) had no binding signature at the time of the audit (2023-05-19), meaning that any binding signature that was present had already expired. In other words: half of all keys used to sign on PyPI since 2020 are already expired. This strongly suggests that nobody is attempting to verify signatures from PyPI on any meaningful scale.

Then, on the algorithm and parameter sides¹²:

Primary keys:

“Effective”¹³ keys:

Or again, as pretty charts:

First, the “good” parts:

1. While normally a bad choice, RSA is literally the best you can do in terms of standard¹⁴ asymmetric signing algorithms in PGP. Over two thirds of keys used to sign on PyPI are using it, and they’re using reasonable¹⁵ key sizes (4096 and 3072).

Then, the meh:

1. A sizeable minority (20% of effective keys, and 17% of primary keys) are RSA-2048. NIST considers RSA-2048 to be equivalent to roughly 112 bits of security¹⁶, and does not recommend its use on data that’s expected to have a security life of 15 years…starting in 2015. That means that PyPI-hosted signatures against RSA-2048 keys have roughly 7 years of “shelf life” in them. Version turnover in packaging ecosystems has accelerated over the last decade; let’s hope that applies here too!

2. Some enterprising people are on the “bleeding edge”: they’re using EdDSA and a few different ECDSA curves. It’s hard to say whether this is good or bad: it’s good in the sense that these are almost certainly better than anything offered by strictly RFC 4880 PGP implementations, but pointless in the sense that support for verifying these signatures is limited¹⁷ to just a few clients. It’s also probably pointlessly slow (for P-521 and brainpoolP512r1 in particular).

And finally, the insane:

• Roughly 5% of all keys used to sign for packages on PyPI are DSA. The majority of those are DSA-1024, which is roughly equivalent in strength to RSA-1024. DSA of any size is already very bad, and DSA-1024 is well outside of any acceptable safety margin for signatures in 2023, much less 2020 or even 2010.

• RSA-4064 and RSA-4032. I have no idea why anyone would do this¹⁸. Maybe some misguided attempt to calculate a precise security margin, or a misreading of someone else’s recommendations?

• One of the RSA-2048 keys has a public exponent of 41, rather than 65537 (which every other RSA key in the dataset uses). Again, I have no idea why anyone would do this: it’s pointlessly slower and opens up padding concerns that e = 65537 is resilient against.

Takeaways

To summarize: of just the PGP signatures uploaded to PyPI in the last three years:

• A small handful (26) are so malformed or old that they can’t be correlated to a key ID;
• Of those that can be correlated to a key ID, nearly a third cannot be discovered through the PGP ecosystem’s intended key discovery mechanism;

• Of the remaining discoverable keys:

  • Nearly half (49%) have no active binding signature when retrieved from a keyserver, giving them indefinite (at the absolute best) identity properties.

  • A sizeable minority (20%) are using weak RSA keys with less than a decade before NIST considers them insecure.

  • A smaller but still appreciable minority (5%) are using DSA-1024 keys, which have been considered insecure for well over a decade.

By all rights, these numbers represent the best possible case for PGP signatures on PyPI. Expanding the audit to 2015 or even earlier would likely reveal far worse practices.

In one sense, none of this is a problem: the breadth and depth of issues here suggests that nobody (thankfully!) is actually relying on these signatures, and the continued presence of new signatures on PyPI is primarily a vestige of forgotten automation and outdated tutorials.

On the other hand, these results present a strong case against attempting to “rehabilitate” PGP signatures for PyPI, or any other packaging ecosystem: all evidence points to end users (i.e., signers) being unable¹⁹ to distinguish between the “good” and “bad” parts of PGP, much less use them at all (e.g. keyservers).

So, for final conclusions:

• Given how broken the PGP signatures and keys present on PyPI are, it’s unlikely that anybody is currently doing wide-scale verification against them.
• If anybody is (and I’d be interested to hear if you are!), then it’s almost certainly inadvisable: “verifying” these signatures is, on average, likely to provide a false degree of confidence in their value.

As with previous posts, I’ve tried to make my steps and data reproducible, and have checked them all into this repo. I welcome any discoveries of mistakes I’ve made, as well as any attempts to improve the overall detail or fidelity of the results!