Oct 23, 2019 Tags: devblog
Travis CI is a wonderful service, and has been a cornerstone of continuous integration in the open source community for years. It has also, unfortunately, stagnated for several years and is currently owned by a private equity firm that appears to have no interest in either new development or maintenance1.
New projects should probably use other CI services, but there are a decent number of projects out there (including projects that I maintain) for which changing CI providers isn’t a high priority. This post covers patching a historical deficiency in Travis’s stage and matrix behavior for projects that would like pipeline-style CI/CD without having the time or resources to switch to a new service.
Many CI/CD configurations are diamond-shaped:
Travis CI’s matrix configuration is well suited to either #2 or #1 and #3, but chokes when tasked with both.
The “right” way to define a variable number of tasks at each point in the CI/CD pipeline is to use Travis’s stages:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
jobs:
include:
- stage: lint
language: minimal
dist: bionic
addons:
apt:
packages:
- clang-format-8
script:
- make lint
- stage: build
os: linux
language: cpp
compiler: clang
script:
- make
- stage: build
os: linux
language: cpp
compiler: gcc
script:
- make
- stage: build
os: osx
language: cpp
compiler: clang
script:
- make
- stage: deploy
os: linux
language: minimal
script:
- make deploy
- stage: deploy
os: osx
language: minimal
script:
- make deploy
This is error-prone and repetitive: it only takes me missing one member of any stage to subtly break the pipeline and introduces visual clutter that YAML is ill-suited to deal with.
YAML has a reputation for complexity and surprising functionality. These are normally negative qualities in a configuration language (especially one known for human comprehensibility). Our present situation is one where where the opposite is true: we’re going to use two of YAML’s unusual and lesser-known features (anchors and extensions) to deduplicate our stage configuration.
Anchors bear a striking similarity to C’s addressing syntax, and behave intuitively if you’re familiar with the former:
1
2
foo: &bar baz
bar: *bar
is expanded as:
1
2
foo: baz
bar: baz
Extensions allow an anchored blob of YAML to be injected:
1
2
3
4
5
blob: &blob
bar: baz
foo:
<<: *blob
is the same as:
1
2
3
4
5
blob:
bar: baz
foo:
bar: baz
Put together, we can use references and anchors to gracefully compose our stages. Underscores are used for the key and anchor to minimize the chance of a namespace conflict, should Travis attempt to either interpret a key name or append additional YAML to a user’s configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
_build: &_build
stage: build
language: cpp
script:
- make
_deploy: &_deploy
stage: deploy
language: minimal
script:
- make deploy
jobs:
include:
- <<: *_build
os: linux
compiler: clang
- <<: *_build
os: linux
compiler: gcc
- <<: *_build
os: osx
compiler: clang
- <<: *_deploy
os: linux
- <<: *_deploy
os: osx
We can even put our addons into the common build fragment, since Travis will only intepret the keys particular to each stage’s configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
_build: &_build
stage: build
language: cpp
addons:
apt:
packages:
- libfoo
- libbar
- libbaz
homebrew:
packages:
- libfoo
- libbar
- libbaz
script:
- make
# Use *_build as above...
Check out pe-parse’s build for a decent example of this.