Dockerizing your CI/CD Pipeline

Paul Dragoonis

  • From Glasgow (a weegie :-)
  • Full-Stack Software Consultant
  • Quality Assurance Engineer
  • @dr4goonis / paul@dragoonis.com

Analyse and influence on quality of

  • code, environments, tech stack
  • BDD, DDD, TDD
  • continuous integration pipelines
  • deployment / rollback quality
  • security / scaling

Today

  • Continuous Deployment vs Delivery
  • Building a strong CI pipeline
  • Tips and Tricks with Jenkins
  • Dockerizing your CI pipeline
  • Outline benefits of dockerizing your dev env / pipeline

Continuous Deployment vs Delivery

  • Depends on your customer / product
  • Depends on your rate of change
  • Depends on your team
  • Depends on your infrastructure
  • Dockerizing your CI pipeline

Building a strong CI pipeline

What is a pipeline?

changes / builds / results moving through a series of broken down steps from one side to the other.

Triggering Builds

  • SCM triggered builds
  • parameterised builds - passing down branch name

What to put in your pipeline

Setup system - pull down code

  • Pull down dependencies (More than one repo? Jenkins MultipleSCM Plugin)
  • or
  • parameterised builds - passing down branch name

Quick initial feedback

  • lint checking
  • unit tests

Perform thorough testing

  • integration tests
  • acceptance
  • end-to-end testing

Linking your pipeline together

Triggers

(sequencing of the pipeline)

  • downstream triggers - what to trigger next
  • upstream triggers - what to trigger before running current one

Managing the Pipline

Build Pipeline Plugin

Build Pipeline Plugin

Delivery Pipeline Plugin

Delivery Pipeline Plugin

CloudBees Folders Plugin

Jobs Maintenance

CloudBees Folders Plugin

JobDSL Plugin


job('PROJ-unit-tests') {
    scm {
        git('git://github.com/jenkinsci/job-dsl-plugin.git')
    }
    triggers {
        scm('*/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}
              

JobDSL Plugin


job('PROJ-release') {
    scm {
        git('git://github.com/jenkinsci/job-dsl-plugin.git')
    }
    // no trigger
    authorization {
        // limit builds to just Jack and Jill
        permission('hudson.model.Item.Build', 'jill')
        permission('hudson.model.Item.Build', 'jack')
    }
    steps {
        maven('-B release:prepare release:perform')
        shell('cleanup.sh')
    }
}
              

JobDSL Plugin



freeStyleJob(String name, Closure closure = null)

buildFlowJob(String name, Closure closure = null)

ivyJob(String name, Closure closure = null)

matrixJob(String name, Closure closure = null)

mavenJob(String name, Closure closure = null)

multiJob(String name, Closure closure = null)

workflowJob(String name, Closure closure = null)

multibranchWorkflowJob(String name, Closure closure = null)
              

JobDSL Plugin


folder('project-a')

deliveryPipelineView('project-a-deliver-pipeline')

buildPipelineView('project-a-build-pipeline')

buildMonitorView('project-a-build-monitor')
              

Problems with standlone Jenkins

Scaling is hard, one slave per machine

One integration environment for your platform, installed on jenkins host

Jenkins env inconsistent from staging/prod

Docker to the rescue

What is docker?

What is docker?

Docker on different OS

Docker Hub (official registry)

Running an image from docker hub


    docker pull image_name:tag
    docker pull nginx:latest
    docker pull node:5.6
    docker pull mysql-server:5.7
              

Making your own image

Dockerfile
FROM ubuntu:14.04

RUN apt-get update && apt-get install -y python python-dev postgresql

USER postgres

RUN /etc/init.d/postgresql start && createdb -T template0 -E UTF8 my_database

USER root

EXPOSE 8000
              

Building your image


docker build -t falkirk_db_image
            

Docker compose (container orchestration)

docker-compose.yml
database:
  image: falkirk_db_image

cache:
  image: redis:latest
  ports:
    - "6379:6379"

web_app:
  build: .
  dockerfile: path/to/my/apps/Dockerfile
  ports:
      - "8080:80"
  links:
      - database
      - redis

            

Running your containers

docker-compose.yml
                docker-compose up
                docker-compose up -d  <--- daemon mode
            

Back to Jenkins land

Docker Jenkins Plugin

Docker Jenkins Plugin

connects to any host running docker

creates a jenkins agent, runs job(s)

spins down jenkins agent

CI Workflow with Docker

CI Workflow with Docker

Spinning up containers for testing


    docker-compose -f docker-compose.ci.testing.yml up -d
    bin/run-test-suite-in-containers.sh
            

Build app image out of green test run


    docker build -t falkirk_lug_app:${BUILD_NUMBER}    <-- jenkins var
            

CI Workflow with Docker

CI Workflow with Docker

Building with dynamic tag


    docker build -t my-registry.com/falkirk_lug_app:${BUILD_NUMBER}
    docker build -t my-registry.com/falkirk_lug_app:JOB-34545
    docker build -t my-registry.com/falkirk_lug_app:3ecedfd8

    docker push my-docker-registry.company.com/falkirk_lug_app:JOB-34545
            

Pushing to private docker registry


    docker push my-docker-registry.company.com/falkirk_lug_app:JOB-34545
            

Registries overview

CI Workflow with Docker

Build promotion

Overview of docker pipeline

Thank you Falkirk!