Continuous Integration has never been so easy

Running tests in a shell step

Script used to run tests

Setup

1. Install Jenkins CI in a server
2. Listen to repository changes with GitHub Pull Request Builder or Generic Webhook Trigger
3. Set up the job(s)

Pros: free and customisable

Install it, create the jobs, and adapt them to your team's workflow

Pros: temporary testing environments

Cons

• Lot of setup, tweaking, and server maintenance required
• It's tricky to keep the job implementation under version control

The big con: software is free, time isn't

• As a self-hosted option, we dive into a real case of the costs of maintaining CI tooling with Jenkins

Assumptions

• Eight developers
• 10+ repositories
• 4 jobs per pull request, 18 minutes total CPU time
• 8 PRs / day, testing 24 total commits
• Team runs some tests locally, reducing CI load due to reworked tests

EC2 as a VPS

• Easiest to understand and setup
• The server is always there adding to your bills
• Activity spikes (like the end of a sprint)
• Autoscaling is a lie

Engineering costs

Juampy spent:

• 116 hours over 15 months
• Time entries logged with keywords Jenkins, CI, or Continous (no migrations!)
• Literally hundreds to thousands of dollars per month
• Doesn't include client team hours which was all of the infrastructure and hosting

Knowledge pool

Your team presumably knows Drupal, JavaScript, programming. You might know Jenkins, but is it your job? What about after the initial setup?

The costs of hosting Jenkins efficiently are directly related to the expertise of your devops staff; you may pay more for a Jenkins expert than the hosting costs of an unoptimized architecture.

Sample

Open it at GitHub

              unit_kernel_tests: &unit_kernel_tests
  <<: *defaults
  steps:
    - checkout
    - *copy_robo
    - restore_cache: *restore_cache
    - run:
        name: Run PHPUnit tests
        command: robo job:run-unit-tests
    - store_test_results:
        path: /var/www/html/artifacts/phpunit
    - store_artifacts:
        path: /var/www/html/artifacts
    - save_cache: *save_cache
              
            

Setup

1. Add a .circleci/config.yml file to the repository
2. Authenticate with your GitHub account at https://circleci.com
3. Allow CircleCI to watch for repository changes

Pros: docker-compose-style environments

              defaults: &defaults
  docker:
    - image: juampynr/drupal8ci:latest

    - image: selenium/standalone-chrome-debug:3.7.1-beryllium

    - image: mariadb:10.3
      environment:
        MYSQL_ALLOW_EMPTY_PASSWORD: 1
              
            

Pros: SSH access to job environments

Pros: Run jobs locally

Cons

• The free service only allows one concurrent job at a time
• Not as customizable as a Jenkins setup

Pros: Costs

• ~50 hours from "we have never used Circle" to done
• Very minimal maintenance - a few hours twice per year for Drupal core upgrades
• Free to ~250 USD / month, scales with your org

.travis.yml

View full file

              cache:
  directories:
    - $HOME/.composer/cache/files
services:
  - docker
env:
  matrix:
    - JOB=job:check-coding-standards
    - JOB=job:run-unit-tests
    - JOB=job:run-behat-tests
script:
  - vendor/bin/robo $JOB
              
            

Cons

• Coverage reports require a third party service like Coveralls
• Supports Docker but needs extra setup
• No Command Line Interface to run jobs locally

But at what cost?

"Tests are too hard"

"We don't trust the tests"

"We will spend more time on tests than fixing bugs"

"We need to deliver"

Evaluating a project for CI

Six months or six years?

Growing business or changing business?

Are you continuously stretching your code base to be larger, or refining and replacing the same code, over and over?

Ease of updates

Can you easily deploy updates, or is software being distributed to far-flung teams?

Environmental Conditions

Where is your code running? One site and environment, or many?

QA Planning

Regression testing is the biggest pain point. Developers as QA can be a great model.

Developers

"Tests are too hard"

Tests are tools and APIs that teach you the underlying systems.

QA

"We don't trust the tests"

If you have a dedicated QA team, they need to be completely integrated with development.

Project Managers

"We will spend more time on tests than fixing bugs"

Be tactical about what to test. Trust your developers. Track your hours.

Product Owners

"We need to deliver"

Continuous integration leads to continuous delivery.