ENOSUCHBLOG

Programming, philosophy, pedaling.


Flexible Travis CI stages with YAML references

Oct 23, 2019

Tags: devblog

Travis CI is a wonderful service, and has been a cornerstone of continuous integration in the open source community for years. It has also, unfortunately, stagnated for several years and is currently owned by a private equity firm that appears to have no interest in either new development or maintenance1.

New projects should probably use other CI services, but there are a decent number of projects out there (including projects that I maintain) for which changing CI providers isn't a high priority. This post covers patching a historical deficiency in Travis's stage and matrix behavior for projects that would like pipeline-style CI/CD without having the time or resources to switch to a new service.

Problem statement

Many CI/CD configurations are diamond-shaped:

  1. Target-independent jobs like automated format checking, static analysis, and linting run first and only (need to) run on one configuration
  2. Unit tests run on every configuration in the matrix (potentially with holes or allowed failures)
  3. Deployment or asset publication happens on select configurations (e.g., after non-debug builds)

Travis CI's matrix configuration is well suited to either #2 or #1 and #3, but chokes when tasked with both.

YAML to the rescue

The "right" way to define a variable number of tasks at each point in the CI/CD pipeline is to use Travis's stages:

jobs:
  include:
    - stage: lint
      language: minimal
      dist: bionic
      addons:
        apt:
          packages:
            - clang-format-8
      script:
        - make lint
    - stage: build
      os: linux
      language: cpp
      compiler: clang
      script:
        - make
    - stage: build
      os: linux
      language: cpp
      compiler: gcc
      script:
        - make
    - stage: build
      os: osx
      language: cpp
      compiler: clang
      script:
        - make
    - stage: deploy
      os: linux
      language: minimal
      script:
        - make deploy
    - stage: deploy
      os: osx
      language: minimal
      script:
        - make deploy

This is error-prone and repetitive: it only takes me missing one member of any stage to subtly break the pipeline and introduces visual clutter that YAML is ill-suited to deal with.

YAML has a reputation for complexity and surprising functionality. These are normally negative qualities in a configuration language (especially one known for human comprehensibility). Our present situation is one where where the opposite is true: we're going to use two of YAML's unusual and lesser-known features (anchors and extensions) to deduplicate our stage configuration.

Anchors bear a striking similarity to C's addressing syntax, and behave intuitively if you're familiar with the former:

foo: &bar baz
bar: *bar

is expanded as:

foo: baz
bar: baz

Extensions allow an anchored blob of YAML to be injected:

blob: &blob
  bar: baz

foo:
  <<: *blob

is the same as:

blob:
  bar: baz

foo:
  bar: baz

Put together, we can use references and anchors to gracefully compose our stages. Underscores are used for the key and anchor to minimize the chance of a namespace conflict, should Travis attempt to either interpret a key name or append additional YAML to a user's configuration:

_build: &_build
  stage: build
  language: cpp
  script:
    - make

_deploy: &_deploy
  stage: deploy
  language: minimal
  script:
    - make deploy

jobs:
  include:
    - <<: *_build
    os: linux
    compiler: clang
    - <<: *_build
    os: linux
    compiler: gcc
    - <<: *_build
    os: osx
    compiler: clang
    - <<: *_deploy
    os: linux
    - <<: *_deploy
    os: osx

We can even put our addons into the common build fragment, since Travis will only intepret the keys particular to each stage's configuration:

_build: &_build
  stage: build
  language: cpp
  addons:
    apt:
      packages:
        - libfoo
        - libbar
        - libbaz
    homebrew:
      packages:
        - libfoo
        - libbar
        - libbaz
  script:
    - make

# Use *_build as above...

Check out pe-parse's build for a decent example of this.


  1. Much less Travis's employees