CI best practices

Continuous Integration - best practices

General tips

• keep the builds and tests fast!
  • true to the XP spirit, turnaround time between check in and feedback (build successful/failed) should be kept to a minimum
  • long running builds and tests should be split up
    • eg. into staged tests: test a few basic things first and only if they are successful, run more detailed tests
• a CI server is not a build artifact repository or storage server!
  • build artifacts in general should be seen as ephemeral
  • if a build artifact is considered a milestone, release candidate, release, etc. it should be copied to a storage server (eg. downloads.eclipse.org)
  • public announcements like "You can get the latest release from our Jenkins instance" are considered to be bad practice

• General tips about testing

Jenkins

Job configuration smells

• build logic leaks to job
• too many build steps
• shell steps
• not under SCM

Source: https://narkisr.github.io/jenkins-scaling/#configuration-smells

Deal with branches

• managing branches from UI results in duplication, create a template project and clone, use programmatic API instead

Job timeouts

• reasonable timeouts should be set to avoid long blocking builds (several hours or days)
  • Freestyle Job
    • Job config -> Build Environment -> Abort the build if it's stuck
      
  • Pipeline Job

pipeline {
  options {
      timeout(time: 30, unit: 'MINUTES') 
  }
  stages { .. }
  // ..
}

Periodical builds (nightly builds)

• instead of building a job every night at 12am whether or not code has changed, it's recommended to poll the SCM for changes instead (or even better, trigger jobs from webhooks)
  • Job config -> Build Triggers -> Poll SCM
    • to avoid load spikes and distribute load more evenly, it's recommended to use a cron definition like H H(0-3) * * * (build is triggered sometime between 12AM and 3AM)

Disk usage

Builds and tests can consume large amounts of data while checking out Git repositories, downloading Maven artifacts and compiling multi-OS, multi-arch, multi-bitness artifacts, etc.
Disk space is cheap, but not free (yet). Therefore disk usage should be reasonable.

Here is a rough guideline for JIPPs:

• up to 50GB of disk space is considered to be fine.
• 50-100GB is mostly considered to be OK, if it has no impact on other projects.
• upwards of 100GB is considered to be excessive and you should check your job structure/configurations

As always there are exceptions to that rule. If a project has good reasons to use a lot of disk space this will be respected.
If not, you will be notified and ultimately blamed publicly.

Build history

• a build history is a good indicator to track the health/stability/test coverage, etc. of your project
• since the build history can take up considerable large amounts of disk space it should be kept at bay
  • it's nice to see the last 500 builds, but in many cases 10-20 builds have nearly the same value to see trends
  • in most cases there is no benefit in keeping more than a few builds with their artifacts, especially if the size is in the 500MB+ range
    • The relevant setting in Jenkins job configurations is Max # of builds to keep with artifacts, this should be kept between 2 and 5, depending on the size of the artifacts (bigger artifacts -> keep less builds)

Console logs

• console logs are important, but if all debug levels are dialed to 11, logs can become quite big (100s of MB)
  • browsers in general don't like huge console logs
  • try to find a balance between huge logs and time spent to run a build again with different logging levels if something failed (maybe a log level parameter can help)

Workspace

• workspaces should be cleaned before the build starts to avoid old/changed files influencing new builds
  • Job config -> Build Environment -> Delete workspace before build starts
• workspaces can also be cleaned when a build is done
  • this is mostly useful for jobs with huge workspaces, where accessing the workspace for debugging purpose is only rarely required
  • Job config -> Post-build Actions -> Delete workspace when build is done

Concurrent builds
Please keep in mind, that if you use the option "Execute concurrent builds if necessary", each build runs in its own workspace. This will multiply the disk usage.

Keep this build forever

• this option should only be used in special circumstances and only for a very limited period of time
• this option should not be used to store the build artifacts (e.g. of a special build, like a release) for an extended period of time
  • instead the build artifacts should be copied/moved to the project's directory on download.eclipse.org

Old build jobs

• best case scenario: all your build configurations are stored in a SCM and you don't have to keep extra job configurations
• if a build job is deprecated or archived, consider deleting the workspace to save disk space