Dockerizing your CI/CD Pipeline

Paul Dragoonis

  • From Glasgow (a weegie :-)
  • Full-Stack Software Consultant
  • Quality Assurance Engineer
  • @dr4goonis / paul@dragoonis.com

Analyse and influence on quality of

  • code quality
  • environmental quality (local, staging, prod)
  • Behavioural-driven-development
  • continuous integration pipelines
  • security / scaling
  • continuous deployment / rollback quality

Automation

ps aux | grep java | grep -v grep | awk '{print $2}' | xargs kill

while read in; do host "$in"; done < sites.txt | grep -iv "GOOGLE" | grep -E '1\.2\.3\.4|5\.6\.7\.8' | sed -e 's/has\ address\ 216.70.91.72//' | sed -e 's/has\ address\ 94.23.33.92//' | while read sites; do curl -sL -w "%{http_code} %{url_effective}\\n" "$sites" -o /dev/null; done | grep -ivE '4.*|5.*' | sed -e 's/200//' | sed -e 's/HTTP/http/'

Automation

kill-java.sh

find-google-referrers.sh > google-referrers.txt

cron entry for: order-pizza.sh ;-) <3 selenium

Continuous Integration

repeatable, continual way to automate tasks

gain feedback on those tasks like: metrics, measurements, reports

Continuous Deployment
vs
Continuous Delivery

Continuous Deployment

your pipline builds continuously

but, you decide when you want to push the big red button

Continuous Delivery

your pipline builds continuously

once you have a green build, it promotes and goes to production with no manual intervention

What is a pipeline?

results moving through a series of steps from one side to the other.

What does a pipeline consist of?

  • jobs, lots of jobs!
  • that have their own single responsibility

What to put in your pipeline

  • your code - More than one repo?
    Jenkins MultipleSCM Plugin
  •  
  • Quick initial checking
  • lint checking
  • unit tests

Perform thorough testing

  • integration tests
  • acceptance
  • end-to-end testing
  • security testing
  • stress testing

Linking your pipeline together

Triggers

(sequencing of the pipeline)

  • downstream triggers - what to trigger next
  • upstream triggers - what to trigger before running current one

Kicking off your pipeline

  • Commit triggered pipelines
  • Manually triggered(parameterised) pipelines

Managing the Pipline

Build Pipeline Plugin

Build Pipeline Plugin

Delivery Pipeline Plugin

Delivery Pipeline Plugin

Maintaining your pipeline/jobs

CloudBees folders plugin
group your jobs

JobDSL Plugin


job('PROJ-unit-tests') {
    scm {
        git('git://github.com/jenkinsci/job-dsl-plugin.git')
    }
    triggers {
        scm('*/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}
              

JobDSL Plugin


job('unit-tests') {
    scm {
        git('git://github.com/jenkinsci/job-dsl-plugin.git')
    }
    steps {
        shell('prepare.sh')
        shell('run-unit-tests.sh')
    }
}
              

JobDSL Plugin


folder('project-a')

deliveryPipelineView('project-a-deliver-pipeline')

buildPipelineView('project-a-build-pipeline')

buildMonitorView('project-a-build-monitor')
              

Problems with standlone Jenkins

Scaling is hard, one slave per machine

One environment for your all your jobs

installed directly onto the jenkins host

CI env inconsistent from staging/prod

Docker to the rescue

What is docker?

Docker is an open-source project, which allows you to create lightweight, portable, distributable, self-sufficient containers, of an application with all of its dependencies

Containers running on a single machine all share the same OS/kernel/filesystem so they start instantly and make more efficient use of RAM.

What is docker?

Docker on different OS

Docker Hub (official registry)

Running an image from docker hub


    docker pull image_name:tag
    docker pull nginx:latest
    docker pull node:5.6
    docker pull mysql-server:5.7
              

Making your own image

Dockerfile
FROM ubuntu:14.04

RUN apt-get update && apt-get install -y python python-dev postgresql

USER postgres

RUN /etc/init.d/postgresql start && createdb -T template0 -E UTF8 my_database

USER root

EXPOSE 5432
              

Making your own image

Dockerfile
FROM php7.0-cli:latest

RUN apt-get update && apt-get install -y nginx

USER www-data
ENTRYPOINT nginx -g 'daemon off;'
EXPOSE 80
              

Building your image


docker build -t ed_techmeetup:1
            

Docker compose (container orchestration)

docker-compose.yml

application:
  build: .
  dockerfile: path/to/my/apps/Dockerfile
  ports:
      - "8080:80"

database:
  image: ed_techmeetup:1

cache:
  image: redis:latest
  ports:
    - "6379:6379"
            

Running your containers


                docker-compose up  <--- you will see all logs of all containers

                docker-compose up -d  <--- daemon mode, runs in background

                docker-compose logs <--
            

Back to Jenkins land

Docker Jenkins Plugin

Docker Jenkins Plugin

connects to any host running docker

creates an isolated environment for your jobs

runs your jobs inside it

spins down isolated environment

Build app image out of green test run


docker build -t my-registry.com/ed_techmeetup:JOB-${BUILD_NUMBER} <- jenkins var
docker build -t my-registry.com/ed_techmeetup:JOB-34545
            

CI Workflow with Docker

Integration testing with your containers


    cp docker-compose.ci.testin.yml docker-compose.yml

    docker-compose -d up

    run-integration-test-suite.sh
            

Pushing to private docker registry


docker push my-docker-registry.company.com/ed_techmeetup:JOB-${BUILD_NUMBER}
            

Overview of docker pipeline

Lessons learned about dockerizing pipelines

Have lots of cleanup tasks, disk space fills up quickly

You can build and run production images on your localhost very easily

Thank you Edinburgh!