Getting Started - Create a Basic Pipeline
Continuous Integration with Codefresh
In this tutorial we will setup a Continuous Integration pipeline within Codefresh using an example application. You will learn:
- How to connect your Git repository
- How to connect your Docker registry
- How to build a Docker image from the source code
- How to push a Docker image to your registry
- How to run unit tests for your application
- How to publish your image to Dockerhub
Codefresh is the fastest way to get from your source code to a Docker image. Codefresh allows you to create a Docker image from its friendly UI without any local Docker installation (Docker building as a service).
You can store the resulting image on a public or private Docker registry that your organization already uses, or any other Docker registry service that you connect to your Codefresh account.
Codefresh also has built-in support for unit and integration testing allowing you to only push Docker images that pass your testing suite. Finally, you can add annotations to your Docker images to better track your releases (e.g. you can mark a Docker image with an annotation that shows a successful unit test run).
You can use either the sample application we provide here to follow along or create your own Docker based example if you prefer (don’t forget to write unit tests).
Prerequisites for this tutorial
For this tutorial you will need:
- A free GitHub account
- A free Codefresh account
- An account to a Docker registry service (e.g. GitHub )
- The source code of the sample application
- (Optional) an account to Dockerhub if you also want to make your image public
We also assume that you are familiar with Docker and the build/run workflow it supports. Your applications should already come with their own Dockerfiles. If not, then read the official documentation first.
Apart from Dockerhub, you can use the registry that comes with your cloud account (Amazon, Azure, Google) or you can use any other compliant service such as Quay, Treescale, Canister.io etc. GitHub also offers a container registry with each account.
The sample application can be found at https://github.com/codefresh-contrib/python-flask-sample-app. To bring the source code to your own account you need to “fork” the repository by clicking the respective button at the top right part of the page.
After some brief time, the repository should appear in your own GitHub account. Now you are ready to start building code with Codefresh!
Codefresh supports GitLab, Bitbucket and Azure GIT repositories apart from GitHub. The same principles presented in this tutorial apply for all Git providers.
Continuous Integration with Codefresh
First, let’s look at an overview of the process that we will create:
The diagram above shows a full Continuous Integration pipeline for the sample application. Starting from left to right the critical path is:
- Codefresh connects to GitHub and checks out the source code of the application.
- Codefresh uses the Dockerfile of the application to create a Docker image.
- Unit tests are run in the same Docker image to verify the correctness of the code
- The Docker image is stored in your private Registry
- The Docker image is also pushed to Dockerhub (optional).
The sample application that we are using is a Python/Flask project with the following key points
- It already has its own Dockerfile in the root of the repository.
- It has unit tests.
Creating a Docker Image
We will start by focusing on the first part of the pipeline overview, the creation of a Docker images.
Docker images play a central role in Codefresh pipelines. They are the basic building blocks that serve as the link between what your source code is producing and what gets deployed. If your own application is not “dockerized” yet, you need to create a Dockerfile for it first, before moving it into the Codefresh infrastructure.
Because all Codefresh capabilities are based on Docker images, Docker is also serving as an abstraction layer over any the implementation language of your source code. Codefresh can work with projects written in Ruby, Python, Java, Node or any other programming language as long as they produce a Docker image. Docker images are a first class citizen in Codefresh pipelines and not just an afterthought.
Connecting your Docker registry to Codefresh
Before we create the pipeline you need to connect at least one Docker registry in your Codefresh account. The pipeline will use this registry to store the Docker image of the application.
Go to your Registry Configuration screen and connect your Docker registry. Codefresh supports all popular Docker registries as well as any service that follows the registry specification. If you don’t already have a registry we recommend you start with the GitHub Registry.
If you connect multiple Docker registries make sure that you select one as the “default”. This will be used automatically by the pipeline as you will see later in the build stage.
Creating a new project
With the Docker registry connected, we can now start our pipeline.
Codefresh pipelines are grouped under projects. Project names can be anything you want with the most common example being the name of an application where all pipelines in the project are packaging/deploying the different microservices. You can think of projects as “folders/directories” for your pipelines.
Make sure that you have selected Projects from the left sidebar. Then click on the New project button on the top right corner to get started.
Enter a name for your project (e.g.
my-first-project) and choose a sample icon that you like. You can also optionally add tags that will be used for access control (most useful in a organization). For now leave the tags empty.
Click the Create button once you are done. You now have a new project and can start adding pipelines in it.
Creating a new pipeline
Click the New pipeline button in order to create a pipeline.
Enter any name (e.g.
Find your repository from the list and select it. Make sure that the option Add Git commit trigger is selected. This way your pipeline will be automatically launched when a commit happens on this repository.
Click the Create button. Your pipeline was created and you should now see the pipeline editor. Here you can describe what the pipeline will do using Codefresh YAML.
Build the docker image
Codefresh has already created a sample pipeline which we will not use for this tutorial.
You will create a very simple pipeline that checks out the source code and builds a docker image. Delete the existing contents on the editor and paste the following:
This pipeline contains just two steps.
git-clonestep for checking out the code
buildstep for building the docker image AND pushing it to the connected Docker registry.
The clone step is also using some built-in pipeline variables. They instruct the pipeline to checkout the exact code that is described from the commit of the trigger. Don’t worry if the exact details are not clear to you yet.
The build step uses a Dockerfile that is located at the root folder of the project and creates a Docker image with tag
Click the Save button to apply your changes. Then click the Run button to start your pipeline. On the dialog that will appear leave the default selections.
Starting the first build
Once the build is started Codefresh will navigate you to the build progress of the sample application.
The build output is split into sections. Expand the section Building Docker Image and look at the logs. After a while the build should finish with success. All previous runs are in the Builds page from now on.
Running unit tests automatically
Like any well-disciplined project, the sample application comes with its associated unit tests. Running unit tests as part of the build process can validate that the Docker image is indeed correct and satisfies the requested functionality. This is the next step in the build process described at the beginning of this tutorial.
We need to add the tests in the build process. To do this we will get back to the pipeline settings of the application. Click on the name of the pipeline you created in the build log screen so you can edit the yaml file.
The sample application already has unit tests that can be executed with:
pip install pytest && pytest
Edit the pipeline as below:
Here we have added a new freestyle step in its own stage that runs unit tests. Freestyle steps are running custom commands inside docker containers and in this case we run the python command inside the docker image that was just created from the previous step (mentioned by the
Notice that Codefresh also has the capability to run integration tests and get test results as well. Therefore, regardless of the type of tests you employ, Codefresh can accommodate your testing process in a fully automated manner as part of the main build.
This time the build results will contain a new section labeled Running unit tests. It will contain the test output of the application.
This concludes the basic build for the example application. Codefresh offers several more capabilities than the ones shown here. You can read more about pipelines in the YAML documentation.
Storing Docker images in Codefresh
If you have been following along so far, you might already be wondering what happens with the resulting Docker image of each build. The Codefresh build logs show that a Docker image is created after each successful build. Where does this image go?
Codefresh has the unique feature where the build step automatically pushes the image to your default Docker registry! All the images that we have created so far, are stored in the registry you connected in the previous section.
If you only use a single registry your pipeline is now complete.
You can inspect all your images from your previous builds by clicking on Images on the left panel. A list of Docker images will appear sorted starting from the most recent. This dashboard is getting live information from your connected Docker registry.
Among the information shown, you can clearly see:
- What is the Git Branch that created this image
- What is the Git Hash that contained the last commit
This information can help you to easily correlate the changes that exist in each Docker images, which is very important knowledge when it comes to deployments (explained in detail in the next tutorial).
If you click on a Docker image you will get many more details about it including a timeline of the labels for this Docker image. You also have the ability to enter custom comments that describe any event that you consider important. Codefresh really shines when it comes to annotating your Docker images with metadata. For more details read the section Annotations.
Codefresh also includes a graphical display of all the layers contained in the Docker image. This can help you identify big layers in your build process and hopefully give you some pointers on how to reduce the size of your deployment binaries.
The built-in Docker image storage is very helpful on its own, but it becomes even more impressive when it is coupled with the capability to use it as a basis for temporary demo environments, as we will see in the next section.
Uploading Docker images to Dockerhub
Using the default Docker registry for automatic pushes is great for simple pipelines. However, sometimes you might also want to push your Docker image to a second registry or make it public in Dockerhub.
Since the build step creates a Docker image and pushes it at the same time, Codefresh also provides a push pipeline step that just pushes an existing Docker image.
Docker images are one of the central concepts in Codefresh pipelines as everything revolves around them. Powerful Codefresh pipelines can be created by using Docker images as build tools, so it is perfectly normal if you manage a large number of images which are not strictly packaged applications. You may create Docker images that contain building or deployment tools and are used as part of the build process instead of the build result.
For the purposes of this tutorial we will also push our sample application to DockerHub which is the free public Docker hosting registry of Docker Inc. You need to create a free account with the service first and note down your username and password. In your own projects you can use any other external registry you wish.
Note that Docker.io only allows you to push images that are tagged with your username. If you have a choice, create a Dockerhub account with the same username that you have in Codefresh. If not, you need to change the Docker image created to match your username.
Once you create your Docker Hub account, go to your Account Configuration, by clicking on Account Settings on the left sidebar of the Codefresh page. On the first section called Integrations click the Configure button next to Docker Registry. Finally click the Add Registry drop-down menu and select Docker Hub.
Enter your Docker Hub credentials and click the TEST button to verify the connection details. You should see a success message. We have now connected our Docker Hub account to our Codefresh account. Make sure that you note down the Registry Name you used.
To actually use the Docker Hub account in a specific project, go back to your pipeline editor. Change the editor contents to:
We now have added a push step at the end of the pipeline. The image is tagged with the name of the branch.
Click Save to apply your changes and Run to start the pipeline again. In the build logs a new panel will appear that shows the push progress:
Note that this is in addition to default Docker registry of your Codefresh account. After the build is finished the Docker image of the sample application is stored both in the default Docker registry and in Dockerhub.
To verify the latter, you can visit your profile in Dockerhub and look at the image details:
Pushing to Dockerhub is the last step in the build pipeline. Now that we have the basic functionality ready we can see how Codefresh handles Continuous Integration with Pull requests and automatic builds.