Getting Started - Create a Basic Pipeline

Continuous Integration with Codefresh

In this tutorial we will setup a Continuous Integration pipeline within Codefresh using an example application. You will learn:

  • how to connect your Git repository
  • how to build a Docker image from the source code
  • how to use the Codefresh internal registry for Docker images
  • how to run unit tests for your application

Codefresh is the fastest way to get from your source code to a Docker image. Codefresh allows you to create a Docker image from its friendly UI without any local Docker installation (Docker building as a service).

You can store the resulting image on a public or private Docker registry that your organization already uses, or in the built-in Docker storage. Each Codefresh account comes with its own free native Docker registry that works exactly like public registries that you are already familiar with (e.g. Dockerhub).

Codefresh also has built-in support for unit and integration testing allowing you to only push Docker images that pass your testing suite. Finally, you can add annotations to your Docker images to better track your releases (e.g. you can mark a Docker image with an annotation that shows a successful unit test run)

You can use either the sample application we provide here to follow along or create your own Docker based example if you prefer (don’t forget to write unit tests).

Prerequisites for this tutorial

For this tutorial you will need

  • a free Github account
  • a free Codefresh account
  • the source code of the sample application.
  • (Optional) Git and Docker installed on your local workstation
  • (Optional) an account to a Docker registry (e.g. Dockerhub)

We also assume that you are familiar with Docker and the build/run workflow it supports. Your applications should already come with their own Dockerfiles. If not, then read the official documentation first.

The sample application can be found at https://github.com/codefresh-contrib/python-flask-sample-app. To bring the source code to your own account you need to “fork” the repository by clicking the respective button at the top right part of the page.

Forking the example project

Forking the example project (click image to enlarge)

After some brief time, the repository should appear in your own Github account. Now you are ready to start building code with Codefresh!

Codefresh supports Gitlab and Bitbucker repositories apart from Github. The same principles presented in this tutorial apply for all Git providers.

Continuous Integration with Codefresh

First, let’s look at an overview of the process that we will create:

Pipeline Overview

Pipeline Overview (click image to enlarge)

The diagram above shows a full Continuous Integration pipeline for the sample application. Starting from left to right the critical path is

  1. Codefresh connects to Github and checks out the source code of the application
  2. Codefresh uses the Dockerfile of the application to create a Docker image
  3. Unit tests are run in the same Docker image to verify the correctness of the code
  4. The Docker image is stored in the internal Codefresh Registry
  5. The Docker image is pushed to a Docker registry

The sample application that we are using is a Python/Flask project with the following key points

  • It already has its own Dockerfile in the root of the repository
  • It has unit tests

Creating a Docker Image

We will start by focusing on the first part of the pipeline overview, the creation of a Docker images.

Preparing a Docker image

Preparing a Docker image (click image to enlarge)

Docker images play a central role in Codefresh pipelines. They are the basic building blocks that serve as the link between what your source code is producing and what gets deployed. If your own application is not “dockerized” yet, you need to create a Dockerfile for it first, before moving it into the Codefresh infrastructure.

Because all Codefresh capabilities are based on Docker images, Docker is also serving as an abstraction layer over any the implementation language of your source code. Codefresh can work with projects written in Ruby, Python, Java, Node or any other programming language as long as they produce a Docker image. Docker images are a first class citizen in Codefresh pipelines and not just an afterthought.

The example application already comes with its own Dockerfile, making the creation of a Codefresh pipeline very easy. Let’s start by going into the Codefresh dashboard (after creating your account)

Selecting your Git repository

Click the Add new repository button to get started.

Add new repository

Add new repository (click image to enlarge)

You will see a list with all your Github repositories. Locate the one that holds the python-flask-sample-app application and click the Next button.

Select a repository

Select a repository (click image to enlarge)

Selecting the build method

At this point Codefresh will ask you how you want to build your application and it will offer 3 options with the middle-one preselected:

Build method

Build method (click image to enlarge)

The options are:

  1. Use Codefresh.yml file (flexible solution)
  2. Use Dockerfile (easy solution)
  3. Create Dockerfile (guided solution)

The middle option is the easiest to start with and the one that we will select. Codefresh will use the application Dockerfile as the basis for the build and test process. This means that your Docker image describes on its own the steps to package and prepare the application. This is usually the case for interpreted runtimes (Ruby, Python etc).

If your own application is using a compiled language such as Go or Java the Dockerfile should both compile and package the application (hopefully using Multi-Stage builds).

In our example application we already have provided an associated Dockerfile, so just click Select in the middle panel to go forward.

Locating the Dockerfile

In this step Codefresh informs you that it will use the Dockerfile located in the root of the repository and shows you the Docker command it will execute to obtain the Docker image. The defaults are ok for the sample application, so click the Next button to go forward.

Locate Dockerfile

Locate Dockerfile (click image to enlarge)

In this step Codefresh shows you the contents of the Dockerfile it has detected, allowing you to make a final check. Click Create to complete this step of the wizard.

Inspecting the Dockerfile

Inspecting the Dockerfile (click image to enlarge)

The final step asks you which branch you want to use for the main build. Leave the default (master) and click Create Pipeline.

The Git repository is now successfully added into Codefresh. From now on you can find the project in the main Codefresh Dashboard (repositories)

Starting the first build

Codefresh will then present to you the configuration settings for your first pipeline. Again, the defaults are fine for the sample application. Click the Build button and then confirm the next dialog to start your first build with Codefresh!

Starting the build

Starting the build (click image to enlarge)

Congratulations! Your first build with Codefresh is now underway. Codefresh will now start the build process by:

  1. Connecting to Github and fetch the contents of the application
  2. Reading the Dockerfile found in the root directory
  3. Building the docker image, exactly as the Docker build command
  4. Storing the result image into its built-in Docker registry

Build progress and results

Once the build is started you Codefresh will navigate you to the build progress of the sample application.

Monitoring the build

Monitoring the build (click image to enlarge)

The build output is split into sections. Expand the section Building Docker Image and look at the logs. You should see similar output to the one produced locally when we explored the sample application.

After a while the build should be finished with success. All previous runs are in the Builds part from now on.

Build details

Build details (click image to enlarge)

Running unit tests automatically

Like any well-disciplined project, the sample application comes with its associated unit tests. Running unit tests as part of the build process can validate that the Docker image is indeed correct and satisfies the requested functionality. This is the next step in the build process described at the beginning of this tutorial.

Unit tests workflow

Unit tests workflow (click image to enlarge)

We need to add the tests in the build process. To do this we will get back to the pipeline settings of the application. First navigate to the Codefresh dashboard and locate the sample application. Then click on the settings icon.

Changing build settings

Changing build settings (click image to enlarge)

The pipeline settings screen will appear. Codefresh has built-in support for unit testing. As part of the build process Codefresh already has a separate section labeled unit tests. Enter the following line:

python setup.py test

These are the expected command for the sample application, so feel free to replace them with your own unit test command.

Adding Unit tests

Adding Unit tests (click image to enlarge)

Notice that Codefresh also has the capability to run integration tests as well. Therefore, regardless of the type of tests you employ, Codefresh can accommodate your testing process in a fully automated manner as part of the main build.

Once you are done with the unit test lines, click first the Save button to apply your changes and then the Build button to restart the build.

This time the build results will contain a new section labeled Running unit tests. It will contain the test output of the application.

Unit tests results

Unit tests results (click image to enlarge)

This concludes the basic build for the example application. Codefresh offers several more capabilities than the ones shown here. If you have an application that has a more complex build process than a single Dockerfile, you should instead spend some time to read the YAML documentation.

Storing Docker images in Codefresh

If you have been following along so far, you might already be wondering what happens with the resulting Docker image of the each build. The Codefresh build logs show that a Docker image is created after each successful build. Where does this image go?

Codefresh has the unique feature of offering its own built-in storage for Docker images! All the images that we have created so far, are stored within your Codefresh account.

Automatic storage of Docker images

Automatic storage of Docker images

This storage does not aim to replace the Docker registry you might already be using (such as Dockerhub or a private Docker registry). It instead plays a complementary role allowing you to easily look at recent Docker images, and answering the age-old question of which Git commit is actually the source of a deployment.

You can inspect all your images from your previous builds by clicking on Images on the left panel. A list of Docker images will appear sorted starting from the most recent.

Recent Docker images

Recent Docker images (click image to enlarge)

Among the information shown, you can clearly see:

  • What is the Git Branch that created this image
  • What is the Git Hash that contained the last commit

This information can help you to easily correlate the changes that exist in each Docker images, which is very important knowledge when it comes to deployments (explained in detail in the next tutorial).

If you click on a Docker image you will get many more details about it including a timeline of the labels for this Docker image. You also have the ability to enter custom comments that describe any event that you consider important. Codefresh really shines when it comes to annotating your Docker images with metadata. For more details read the section Annotations

Docker Image timeline

Docker Image timeline (click image to enlarge)

Codefresh also includes a graphical display of all the layers contained in the Docker image. This can help you identify big layers in your build process and hopefully give you some pointers on how to reduce the size of your deployment binaries.

Docker Layer Analysis

Docker Layer Analysis (click image to enlarge)

The built-in Docker image storage is very helpful on its own, but it becomes even more impressive when it is coupled with the capability to use it as a basis for temporary demo environments, as we will see in the next section.

Uploading Docker images to a registry

The built-in Docker image storage by Codefresh is ideal for an overview of your images and quick demos. When it comes to production deployments however, your Docker images should be pushed into your own private or public Docker registry.

Kubernetes will then fetch those Docker images from the registry in a well disciplined manner. Remember that Codefresh will automatically keep images from all your builds. It is best to decide which images are actually worth to be deployed and only push those to a production registry.

Pushing a Docker image

Pushing a Docker image (click image to enlarge)

Docker images are one of the central concepts in Codefresh pipelines as everything revolves around them. Powerful Codefresh pipelines can be created by using Docker images as build tools, so it is perfectly normal if you manage a large number of images which are not strictly packaged applications. You may create Docker images that contain building or deployment tools and are used as part of the build process instead of the build result.

For the purposes of this tutorial we will push our sample application to DockerHub which is the free public Docker hosting from Docker Inc. You need to create a free account with the service first and note down your username and password.

Once you create your Docker Cloud account, go to your Account Configuration, by clicking on Configuration on the left sidebar. Then open the Integration tab and click the Add Registry button.

Docker Hub credentials

Docker Hub credentials (click image to enlarge)

Enter your Docker Hub credentials and click the TEST button to verify the connection details. You should see a success message. We have now connected our Docker Hub account to our Codefresh account.

To actually use the Docker Hub account in a specific project, go to the configuration screen of the sample application, by clicking on the “gear” icon found on the Codefresh dashboard under the project name. This is the same screen that we added the unit tests.

Scroll a bit down and directly under the “Unit test” step you will find the “Push to Registry” Step.

Setting the Docker Registry

Setting the Docker Registry (click image to enlarge)

Select the DockerHub entry for the list (which is the one that we defined in the Codefresh Account Setting screen). That’s it! Now your build will automatically use this Docker Registry as a target. Click the Save button to apply the changes and then the Build button to start another build.

In the build logs a new panel will appear that shows the push progress:

Pushing to Docker Hub

Pushing to Docker Hub (click image to enlarge)

Note that this is in addition to the internal Codefresh Docker storage. After the build is finished the Docker image of the sample application is stored both in the Codefresh internal registry and the public Docker Registry.

To verify the latter, you can visit your profile in Docker hub and look at the image details:

Docker Image details

Docker Image details (click image to enlarge)

Pushing to the Docker Registry is the last step in the build pipeline. Now that we have the basic functionality ready we can see how Codefresh handles Continuous Integration with Pull requests and automatic builds.