Docker Continuous Integration with Jenkins and Codefresh

Many development shops have made a big investment in Jenkins for their CI, but with the growing popularity of containers such as Docker, there is a need for a new kind of CI/CD platform, one that is container-native.

That said, not all of the software components in a typical build process are whole applications that can be packaged and delivered in a Docker container. Instead, they are external components, such as libraries, modules, drivers, etc. For these components we actually need to use a traditional CI server.

The result is a need to combine a traditional CI server, such as Jenkins, for non-containerized components, together with a container-native CI tool for the final application images.

In this post, we will show exactly how you can connect Jenkins to the Codefresh container-native CI, resulting in a continuous integration pipeline for both utility components and containerized applications.

We’ll use an example pipeline composed of two building blocks:

a very simple npm package
an Express.js web application that uses this package

Let’s say the npm package is being developed by one of our dev teams. Whenever they push a change to the git repository, Jenkins gets triggered to perform the following flow:

Jenkins bumps the package version, runs unit tests — and if they pass — publishes the package to npm registry.
Jenkins then triggers Codefresh to rebuild the container image which pulls in the latest npm package version.
If the build is successful, another request to Codefresh is performed to bring up the application for testing.

The npm package we are building performs left-pad string manipulation. In light of the recent npm gate scandal, it seems like a good idea to develop this advanced functionality in-house 🙂

The source code for the package can be found here and it’s dead simple:


module.exports = leftpad;
var pjson = require('./package.json');

function leftpad (str, len, ch) {
  
  console.log(pjson.name+"-"+pjson.version);
  str = String(str);
  var i = -1;

  if (!ch && ch !== 0) ch = ' ';

  len = len - str.length;

  while (++i < len) {
    str = ch + str;
  }
  return str;
}

Now let’s look at how our Codefresh service is defined. The application source code can be found here.

In the source tree there’s a Dockerfile:

FROM node:4.4.3

MAINTAINER Anton Weiss <anton@otomato.link>

LABEL Description=”Serve left-padded strings”

WORKDIR /var/www

ADD . /var/www

CMD npm install && npm start

This defines that we are installing npm packages for every image build. The dependency on the cf-left-pad package is defined in package.json like this:

“cf-left-pad” : “*”

This ensures that the latest version of the package will get pulled in for each ‘npm install’ execution.

You can see the service definition in Codefresh:

Let’s start building our pipeline!

Our first Jenkins job will be called ‘build-cf-left-pad’. We’ll configure it to pull the code from the GitHub repo and to run the following as ‘execute shell’ build step:

npm install ci-npm-publish
npm version patch
export VERSION=`python -c "import json; print(json.loads(open('package.json').read())['version'])"`
echo VERSION=${VERSION} > version.properties
npm-publish --npmuser ${my_npm_user} --npmemail ${npm_emal} --npmpassword ${npmpwd}

This script uses the ci-npm-publish package that allows passing npm registry credentials on CLI for npm publishing.

‘npm version patch’ bumps the package version number. This number is then saved to the version.properties file. And ‘npm-publish’ takes care of the publishing.

The following screenshot shows the build step configuration in Jenkins:

Once the first build job is finished, it triggers another job named ‘call-codefresh’ The purpose of this job is to trigger an image rebuild on Codefresh. To do this, I used Jenkins Http Request plugin, configured like this:

You can see I’m using https://g.codefresh.io/api/builds/${CODEFRESH_SERVICE_ID} API endpoint with my Codefresh access token for authorization.

This triggers a new image build and outputs the unique build id, which Jenkins saves to the buildid.properties file to pass it to the downstream job.

The downstream job is called ‘launch-cf-service’. It now launches the service we just built the image for using the following shell script code:

export SERVICE_SHA=`curl –header “Content-Type: application/json” –header “Accept: application/json” –header “x-access-token: ${CF_TOKEN}”

“https://g.codefresh.io/api/builds” | jq “.builds[] |.[] | select(._id == “”${CF_BUILD}””) | .sha”`

curl -X POST –header “Content-Type: application/json” –header “Accept: application/json”

–header “x-access-token: ${CF_TOKEN}”

-d “{“repoOwner”: “antweiss”,

“repoName”: “left-padder”,

“sha”: ${SERVICE_SHA},

“branch”: “master”,

“repoData”: {

“url”: {

“https”: “https://github.com/antweiss/left-padder.git”

}

}” https://g.codefresh.io/api/runtime/testit

That’s a bit more complicated, so let’s walk through it.

The first command finds the git commit hash of the latest service image build by sending a request to https://g.codefresh.io/api/builds and parsing that with the wonderful jq tool to find the commit that corresponds to the ${CF_BUILD} id we got from the upstream job.

The second command sends a request to the https://g.codefresh.io/api/runtime/testit endpoint with a json that describes the service we want to launch and the git commit of the last image we want to launch it from. This launches the service on Codefresh, and with that, our short pipeline is complete.

Let’s look at it in action.

The build pipeline on Jenkins: