Getting Started - Create a Basic Pipeline

Continuous Integration with Codefresh

In this tutorial we will setup a Continuous Integration pipeline within Codefresh using an example application. You will learn:

  • how to connect your Git repository
  • how to build a Docker image from the source code
  • how to use the Codefresh internal registry for Docker images
  • how to run unit tests for your application

Codefresh is the fastest way to get from your source code to a Docker image. Codefresh allows you to create a Docker image from its friendly UI without any local Docker installation (Docker building as a service).

You can store the resulting image on a public or private Docker registry that your organization already uses, or in the built-in Docker storage. Each Codefresh account comes with its own free native Docker registry that works exactly like public registries that you are already familiar with (e.g. Dockerhub).

Codefresh also has built-in support for unit and integration testing allowing you to only push Docker images that pass your testing suite. Finally, you can add annotations to your Docker images to better track your releases (e.g. you can mark a Docker image with an annotation that shows a successful unit test run).

You can use either the sample application we provide here to follow along or create your own Docker based example if you prefer (don’t forget to write unit tests).

Prerequisites for this tutorial

For this tutorial you will need

We also assume that you are familiar with Docker and the build/run workflow it supports. Your applications should already come with their own Dockerfiles. If not, then read the official documentation first.

The sample application can be found at https://github.com/codefresh-contrib/python-flask-sample-app. To bring the source code to your own account you need to “fork” the repository by clicking the respective button at the top right part of the page.

Forking the example project

Forking the example project (click image to enlarge)

After some brief time, the repository should appear in your own Github account. Now you are ready to start building code with Codefresh!

Codefresh supports Gitlab and Bitbucket repositories apart from Github. The same principles presented in this tutorial apply for all Git providers.

Continuous Integration with Codefresh

First, let’s look at an overview of the process that we will create:

Pipeline Overview

Pipeline Overview (click image to enlarge)

The diagram above shows a full Continuous Integration pipeline for the sample application. Starting from left to right the critical path is:

  1. Codefresh connects to Github and checks out the source code of the application
  2. Codefresh uses the Dockerfile of the application to create a Docker image
  3. Unit tests are run in the same Docker image to verify the correctness of the code
  4. The Docker image is stored in the internal Codefresh Registry
  5. The Docker image is pushed to a Docker registry

The sample application that we are using is a Python/Flask project with the following key points

  • It already has its own Dockerfile in the root of the repository
  • It has unit tests

Creating a Docker Image

We will start by focusing on the first part of the pipeline overview, the creation of a Docker images.

Preparing a Docker image

Preparing a Docker image (click image to enlarge)

Docker images play a central role in Codefresh pipelines. They are the basic building blocks that serve as the link between what your source code is producing and what gets deployed. If your own application is not “dockerized” yet, you need to create a Dockerfile for it first, before moving it into the Codefresh infrastructure.

Because all Codefresh capabilities are based on Docker images, Docker is also serving as an abstraction layer over any the implementation language of your source code. Codefresh can work with projects written in Ruby, Python, Java, Node or any other programming language as long as they produce a Docker image. Docker images are a first class citizen in Codefresh pipelines and not just an afterthought.

The example application already comes with its own Dockerfile, making the creation of a Codefresh pipeline very easy. Let’s start by going into the Codefresh dashboard (after creating your account)

Creating a new project

Codefresh pipelines are grouped under projects. Project names can be anything you want with the most common example being the name of an application where all pipelines in the project are packaging/deploying the different microservices. You can think of projects as “folders/directories” for your pipelines.

Make sure that you have selected Projects from the left sidebar. Then click on the Add project button on the top right corner to get started.

Enter a name for your project (e.g. my-first-project) and choose a sample icon that you like. You can also optionally add tags that will be used for access control (most useful in a organization). For now leave the tags empty.

Click the Create button once you are done. You now have a new project and can start adding pipelines in it.

Creating a new pipeline

Click the Add pipeline button in order to create a pipeline. Enter a name (e.g. basic-build). Make sure that the option Add Git commit trigger is selected.

Create a new pipeline

Create a new pipeline (click image to enlarge)

Find your repository from the list and select it. This way your pipeline will be automatically launched when a commit happens on this repository.

You can also select a template for the pipeline. The Python one is relevant for this project, but we will not use it anyway so feel free to select anything from the list.

Click the Create button. Your pipeline was created and you should now see the pipeline editor. Here you can describe what the pipeline will do using Codefresh YAML.

Build the docker image

You will create a very simple pipeline that checks out the source code and build a docker image. Delete the existing contents on the editor and paste the following:

codefresh.yml

version: '1.0'
stages:
  - checkout
  - package
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: checkout
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: package
    image_name: my-app-image
    working_directory: ./
    tag: v1.0.0
    dockerfile: Dockerfile
      

This pipeline contains just two steps.

The clone step is also using some built-in pipeline variables. They instruct the pipeline to checkout the exact code that is described from the commit of the trigger. Don’t worry if the exact details are not clear to you yet.

The build step uses a Dockerfile that is located at the root folder of the project and creates a Docker image with tag v1.0.0.

Click the Save button to apply your changes. Then click the Run button to start your pipeline. On the dialog that will appear leave the default selections.

Starting the first build

Once the build is started you Codefresh will navigate you to the build progress of the sample application.

Monitoring the build

Monitoring the build (click image to enlarge)

The build output is split into sections. Expand the section Building Docker Image and look at the logs. After a while the build should finish with success. All previous runs are in the Builds part from now on.

Build details

Build details (click image to enlarge)

Running unit tests automatically

Like any well-disciplined project, the sample application comes with its associated unit tests. Running unit tests as part of the build process can validate that the Docker image is indeed correct and satisfies the requested functionality. This is the next step in the build process described at the beginning of this tutorial.

Unit tests workflow

Unit tests workflow (click image to enlarge)

We need to add the tests in the build process. To do this we will get back to the pipeline settings of the application. Click on the name of the pipeline in the build log screen.

The sample application already has unit tests that can be executed with:

python setup.py test

Edit the pipeline as below:

codefresh.yml

version: '1.0'
stages:
  - checkout
  - package
  - test  
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: checkout
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: package
    image_name: my-app-image
    working_directory: ./
    tag: v1.0.1
    dockerfile: Dockerfile
  MyUnitTests:
    title: Running Unit tests
    image: '${{MyAppDockerImage}}'
    stage: test 
    commands:
      - python setup.py test    
      

Here we have added a new freestyle step in its own stage that runs unit tests. Freestyle steps are running custom commands inside docker containers and in this case we run the python command inside the docker image that was just created from the previous step (mentioned by the image property)

Notice that Codefresh also has the capability to run integration tests and get test results as well. Therefore, regardless of the type of tests you employ, Codefresh can accommodate your testing process in a fully automated manner as part of the main build.

This time the build results will contain a new section labeled Running unit tests. It will contain the test output of the application.

Unit tests results

Unit tests results (click image to enlarge)

This concludes the basic build for the example application. Codefresh offers several more capabilities than the ones shown here. You can read more about pipelines in the YAML documentation.

Storing Docker images in Codefresh

If you have been following along so far, you might already be wondering what happens with the resulting Docker image of the each build. The Codefresh build logs show that a Docker image is created after each successful build. Where does this image go?

Codefresh has the unique feature of offering its own built-in storage for Docker images! All the images that we have created so far, are stored within your Codefresh account.

Automatic storage of Docker images

Automatic storage of Docker images

This storage does not aim to replace the Docker registry you might already be using (such as Dockerhub or a private Docker registry). It instead plays a complementary role allowing you to easily look at recent Docker images, and answering the age-old question of which Git commit is actually the source of a deployment.

You can inspect all your images from your previous builds by clicking on Images on the left panel. A list of Docker images will appear sorted starting from the most recent.

Recent Docker images

Recent Docker images (click image to enlarge)

Among the information shown, you can clearly see:

  • What is the Git Branch that created this image
  • What is the Git Hash that contained the last commit

This information can help you to easily correlate the changes that exist in each Docker images, which is very important knowledge when it comes to deployments (explained in detail in the next tutorial).

If you click on a Docker image you will get many more details about it including a timeline of the labels for this Docker image. You also have the ability to enter custom comments that describe any event that you consider important. Codefresh really shines when it comes to annotating your Docker images with metadata. For more details read the section Annotations

Docker Image timeline

Docker Image timeline (click image to enlarge)

Codefresh also includes a graphical display of all the layers contained in the Docker image. This can help you identify big layers in your build process and hopefully give you some pointers on how to reduce the size of your deployment binaries.

Docker Layer Analysis

Docker Layer Analysis (click image to enlarge)

The built-in Docker image storage is very helpful on its own, but it becomes even more impressive when it is coupled with the capability to use it as a basis for temporary demo environments, as we will see in the next section.

Uploading Docker images to a registry

The built-in Docker image storage by Codefresh is ideal for an overview of your images and quick demos. When it comes to production deployments however, your Docker images should be pushed into your own private or public Docker registry.

Kubernetes will then fetch those Docker images from the registry in a well disciplined manner. Remember that Codefresh will automatically keep images from all your builds. It is best to decide which images are actually worth to be deployed and only push those to a production registry.

Pushing a Docker image

Pushing a Docker image (click image to enlarge)

Docker images are one of the central concepts in Codefresh pipelines as everything revolves around them. Powerful Codefresh pipelines can be created by using Docker images as build tools, so it is perfectly normal if you manage a large number of images which are not strictly packaged applications. You may create Docker images that contain building or deployment tools and are used as part of the build process instead of the build result.

For the purposes of this tutorial we will push our sample application to DockerHub which is the free public Docker hosting from Docker Inc. You need to create a free account with the service first and note down your username and password. In your own projects you can use any other external registry you wish.

Note that Docker.io only allows you to push images that are tagged with your username. If you have a choice, create a Dockerhub account with the same username that you have in Codefresh. If not, you need to change the Docker image created to match your username

Once you create your Docker Cloud account, go to your Account Configuration, by clicking on Account Settings on the left sidebar. On the first section called Integrations click the Configure button next to Docker Registry. Finally click the Add Registry drop-down menu and select Docker Hub.

Docker Hub credentials

Docker Hub credentials (click image to enlarge)

Enter your Docker Hub credentials and click the TEST button to verify the connection details. You should see a success message. We have now connected our Docker Hub account to our Codefresh account. Make sure that you note down the Registry Name you used.

To actually use the Docker Hub account in a specific project, go back to your pipeline editor. Change the editor contents to:

version: '1.0'
stages:
  - checkout
  - package
  - test 
  - upload    
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
    stage: checkout
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: package
    image_name: my-app-image
    working_directory: ./
    tag: v1.0.1
    dockerfile: Dockerfile
  MyUnitTests:
    title: Running Unit tests
    image: '${{MyAppDockerImage}}'
    stage: test 
    commands:
      - python setup.py test    
  MyPushStep:
    title: Pushing to Docker Registry
    type: push
    stage: upload
    tag: '${{CF_BRANCH}}'
    candidate: '${{MyAppDockerImage}}'
    image_name: kkapelon/pythonflasksampleapp #Change kkapelon to your dockerhub username
    registry: dockerhub # Name of your integration as was defined in the Registry screen
      

We now have added a push step at the end of the pipeline. The image is tagged with the name of the branch.

Click Save to apply your changes and Run to start the pipeline again. In the build logs a new panel will appear that shows the push progress:

Pushing to Docker Hub

Pushing to Docker Hub (click image to enlarge)

Note that this is in addition to the internal Codefresh Docker storage. After the build is finished the Docker image of the sample application is stored both in the Codefresh internal registry and the public Docker Registry.

To verify the latter, you can visit your profile in Docker hub and look at the image details:

Docker Image details

Docker Image details (click image to enlarge)

Pushing to the Docker Registry is the last step in the build pipeline. Now that we have the basic functionality ready we can see how Codefresh handles Continuous Integration with Pull requests and automatic builds.