Oct 23, 2019 Tags: devblog
Travis CI is a wonderful service, and has been a cornerstone of continuous integration in the open source community for years. It has also, unfortunately, stagnated for several years and is currently owned by a private equity firm that appears to have no interest in either new development or maintenance1.
New projects should probably use other CI services, but there are a decent number of projects out there (including projects that I maintain) for which changing CI providers isn’t a high priority. This post covers patching a historical deficiency in Travis’s stage and matrix behavior for projects that would like pipeline-style CI/CD without having the time or resources to switch to a new service.
Many CI/CD configurations are diamond-shaped:
Travis CI’s matrix configuration is well suited to either #2 or #1 and #3, but chokes when tasked with both.
The “right” way to define a variable number of tasks at each point in the CI/CD pipeline is to use Travis’s stages:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 jobs: include: - stage: lint language: minimal dist: bionic addons: apt: packages: - clang-format-8 script: - make lint - stage: build os: linux language: cpp compiler: clang script: - make - stage: build os: linux language: cpp compiler: gcc script: - make - stage: build os: osx language: cpp compiler: clang script: - make - stage: deploy os: linux language: minimal script: - make deploy - stage: deploy os: osx language: minimal script: - make deploy
This is error-prone and repetitive: it only takes me missing one member of any stage to subtly break the pipeline and introduces visual clutter that YAML is ill-suited to deal with.
YAML has a reputation for complexity and surprising functionality. These are normally negative qualities in a configuration language (especially one known for human comprehensibility). Our present situation is one where where the opposite is true: we’re going to use two of YAML’s unusual and lesser-known features (anchors and extensions) to deduplicate our stage configuration.
Anchors bear a striking similarity to C’s addressing syntax, and behave intuitively if you’re familiar with the former:
1 2 foo: &bar baz bar: *bar
is expanded as:
1 2 foo: baz bar: baz
Extensions allow an anchored blob of YAML to be injected:
1 2 3 4 5 blob: &blob bar: baz foo: <<: *blob
is the same as:
1 2 3 4 5 blob: bar: baz foo: bar: baz
Put together, we can use references and anchors to gracefully compose our stages. Underscores are used for the key and anchor to minimize the chance of a namespace conflict, should Travis attempt to either interpret a key name or append additional YAML to a user’s configuration:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 _build: &_build stage: build language: cpp script: - make _deploy: &_deploy stage: deploy language: minimal script: - make deploy jobs: include: - <<: *_build os: linux compiler: clang - <<: *_build os: linux compiler: gcc - <<: *_build os: osx compiler: clang - <<: *_deploy os: linux - <<: *_deploy os: osx
We can even put our addons into the common build fragment, since Travis will only intepret the keys particular to each stage’s configuration:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 _build: &_build stage: build language: cpp addons: apt: packages: - libfoo - libbar - libbaz homebrew: packages: - libfoo - libbar - libbaz script: - make # Use *_build as above...
Check out pe-parse’s build for a decent example of this.