Getting Started - Create a Basic Pipeline
Continuous Integration with Codefresh
In this tutorial we will setup a Continuous Integration pipeline within Codefresh using an example application. You will learn:
- how to connect your Git repository
- how to build a Docker image from the source code
- how to use the Codefresh internal registry for Docker images
- how to run unit tests for your application
Codefresh is the fastest way to get from your source code to a Docker image. Codefresh allows you to create a Docker image from its friendly UI without any local Docker installation (Docker building as a service).
You can store the resulting image on a public or private Docker registry that your organization already uses, or in the built-in Docker storage. Each Codefresh account comes with its own free native Docker registry that works exactly like public registries that you are already familiar with (e.g. Dockerhub).
Codefresh also has built-in support for unit and integration testing allowing you to only push Docker images that pass your testing suite. Finally, you can add annotations to your Docker images to better track your releases (e.g. you can mark a Docker image with an annotation that shows a successful unit test run).
You can use either the sample application we provide here to follow along or create your own Docker based example if you prefer (don’t forget to write unit tests).
Prerequisites for this tutorial
For this tutorial you will need
- a free Github account
- a free Codefresh account
- the source code of the sample application.
- (Optional) an account to a Docker registry (e.g. Dockerhub)
We also assume that you are familiar with Docker and the build/run workflow it supports. Your applications should already come with their own Dockerfiles. If not, then read the official documentation first.
The sample application can be found at https://github.com/codefresh-contrib/python-flask-sample-app. To bring the source code to your own account you need to “fork” the repository by clicking the respective button at the top right part of the page.
After some brief time, the repository should appear in your own Github account. Now you are ready to start building code with Codefresh!
Codefresh supports Gitlab and Bitbucker repositories apart from Github. The same principles presented in this tutorial apply for all Git providers.
Continuous Integration with Codefresh
First, let’s look at an overview of the process that we will create:
The diagram above shows a full Continuous Integration pipeline for the sample application. Starting from left to right the critical path is:
- Codefresh connects to Github and checks out the source code of the application
- Codefresh uses the Dockerfile of the application to create a Docker image
- Unit tests are run in the same Docker image to verify the correctness of the code
- The Docker image is stored in the internal Codefresh Registry
- The Docker image is pushed to a Docker registry
The sample application that we are using is a Python/Flask project with the following key points
- It already has its own Dockerfile in the root of the repository
- It has unit tests
Creating a Docker Image
We will start by focusing on the first part of the pipeline overview, the creation of a Docker images.
Docker images play a central role in Codefresh pipelines. They are the basic building blocks that serve as the link between what your source code is producing and what gets deployed. If your own application is not “dockerized” yet, you need to create a Dockerfile for it first, before moving it into the Codefresh infrastructure.
Because all Codefresh capabilities are based on Docker images, Docker is also serving as an abstraction layer over any the implementation language of your source code. Codefresh can work with projects written in Ruby, Python, Java, Node or any other programming language as long as they produce a Docker image. Docker images are a first class citizen in Codefresh pipelines and not just an afterthought.
Selecting your Git repository
Make sure that you have selectec Repositories from the left sidebar. Then click the Add new repository button to get started.
You will see a list with all your Github repositories. Locate the one that holds the
python-flask-sample-app application and click the Next button.
Selecting the build method
At this point Codefresh will ask you how you want to build your application and it will offer 3 options with the middle-one preselected:
The options are:
- Use Codefresh.yml file (flexible solution)
- Use Dockerfile (easy solution)
- Create Dockerfile (guided solution)
The middle option is the easiest to start with and the one that we will select. Codefresh will use the application Dockerfile as the basis for the build and test process. This means that your Docker image describes on its own the steps to package and prepare the application. This is usually the case for interpreted runtimes (Ruby, Python etc).
If your own application is using a compiled language such as Go or Java the Dockerfile should both compile and package the application (hopefully using Multi-Stage builds).
In our example application we already have provided an associated Dockerfile, so just click Select in the middle panel to go forward.
Locating the Dockerfile
In this step Codefresh informs you that it will use the Dockerfile located in the root of the repository and shows you the Docker command it will execute to obtain the Docker image. The defaults are ok for the sample application, so click the Next button to go forward.
In this step Codefresh shows you the contents of the Dockerfile it has detected, allowing you to make a final check. Click Create to complete this step of the wizard.
The final step asks you which branch you want to use for the main build. Leave the default (master) and click Create Pipeline.
The Git repository is now successfully added into Codefresh. From now on you can find the project in the main Codefresh Dashboard (repositories)
Starting the first build
Codefresh will then present to you the configuration settings for your first pipeline. Again, the defaults are fine for the sample application. Click the Build button and then confirm the next dialog to start your first build with Codefresh!
Congratulations! Your first build with Codefresh is now underway. Codefresh will now start the build process by:
- Connecting to Github and fetch the contents of the application
- Reading the Dockerfile found in the root directory
- Building the docker image, exactly as the Docker
- Storing the result image into its built-in Docker registry
Build progress and results
Once the build is started you Codefresh will navigate you to the build progress of the sample application.
The build output is split into sections. Expand the section Building Docker Image and look at the logs. You should see similar output to the one produced locally when we explored the sample application.
After a while the build should be finished with success. All previous runs are in the Builds part from now on.
Running unit tests automatically
Like any well-disciplined project, the sample application comes with its associated unit tests. Running unit tests as part of the build process can validate that the Docker image is indeed correct and satisfies the requested functionality. This is the next step in the build process described at the beginning of this tutorial.
We need to add the tests in the build process. To do this we will get back to the pipeline settings of the application. First navigate to the Codefresh dashboard and locate the sample application. Then click on the settings icon.
The pipeline settings screen will appear. Codefresh has built-in support for unit testing. As part of the build process Codefresh already has a separate section labeled unit tests. Enter the following line:
python setup.py test
These are the expected command for the sample application, so feel free to replace them with your own unit test command.
Notice that Codefresh also has the capability to run integration tests as well. Therefore, regardless of the type of tests you employ, Codefresh can accommodate your testing process in a fully automated manner as part of the main build.
Once you are done with the unit test lines, click first the Save button to apply your changes and then the Build button to restart the build.
This time the build results will contain a new section labeled Running unit tests. It will contain the test output of the application.
This concludes the basic build for the example application. Codefresh offers several more capabilities than the ones shown here. If you have an application that has a more complex build process than a single Dockerfile, you should instead spend some time to read the YAML documentation.
Storing Docker images in Codefresh
If you have been following along so far, you might already be wondering what happens with the resulting Docker image of the each build. The Codefresh build logs show that a Docker image is created after each successful build. Where does this image go?
Codefresh has the unique feature of offering its own built-in storage for Docker images! All the images that we have created so far, are stored within your Codefresh account.
This storage does not aim to replace the Docker registry you might already be using (such as Dockerhub or a private Docker registry). It instead plays a complementary role allowing you to easily look at recent Docker images, and answering the age-old question of which Git commit is actually the source of a deployment.
You can inspect all your images from your previous builds by clicking on Images on the left panel. A list of Docker images will appear sorted starting from the most recent.
Among the information shown, you can clearly see:
- What is the Git Branch that created this image
- What is the Git Hash that contained the last commit
This information can help you to easily correlate the changes that exist in each Docker images, which is very important knowledge when it comes to deployments (explained in detail in the next tutorial).
If you click on a Docker image you will get many more details about it including a timeline of the labels for this Docker image. You also have the ability to enter custom comments that describe any event that you consider important. Codefresh really shines when it comes to annotating your Docker images with metadata. For more details read the section Annotations
Codefresh also includes a graphical display of all the layers contained in the Docker image. This can help you identify big layers in your build process and hopefully give you some pointers on how to reduce the size of your deployment binaries.
The built-in Docker image storage is very helpful on its own, but it becomes even more impressive when it is coupled with the capability to use it as a basis for temporary demo environments, as we will see in the next section.
Uploading Docker images to a registry
The built-in Docker image storage by Codefresh is ideal for an overview of your images and quick demos. When it comes to production deployments however, your Docker images should be pushed into your own private or public Docker registry.
Kubernetes will then fetch those Docker images from the registry in a well disciplined manner. Remember that Codefresh will automatically keep images from all your builds. It is best to decide which images are actually worth to be deployed and only push those to a production registry.
Docker images are one of the central concepts in Codefresh pipelines as everything revolves around them. Powerful Codefresh pipelines can be created by using Docker images as build tools, so it is perfectly normal if you manage a large number of images which are not strictly packaged applications. You may create Docker images that contain building or deployment tools and are used as part of the build process instead of the build result.
For the purposes of this tutorial we will push our sample application to DockerHub which is the free public Docker hosting from Docker Inc. You need to create a free account with the service first and note down your username and password. In your own projects you can use any other external registry you wish.
Note that Docker.io only allows you to push images that are tagged with your username. If you have a choice, create a Dockerhub account with the same username that you have in Codefresh. If not, you need to change the Docker image created to match your username
Once you create your Docker Cloud account, go to your Account Configuration, by clicking on Account Settings on the left sidebar. On the first section called Integrations click the Configure button next to Docker Registry. Finally click the Add Registry drop-down menu and select Docker Hub.
Enter your Docker Hub credentials and click the TEST button to verify the connection details. You should see a success message. We have now connected our Docker Hub account to our Codefresh account.
To actually use the Docker Hub account in a specific project, go to the configuration screen of the sample application, by clicking on the “gear” icon found on the Codefresh dashboard under the project name. This is the same screen that we added the unit tests.
Scroll a bit down and directly under the “Unit test” step you will find the “Push to Registry” Step.
Select the DockerHub entry for the list (which is the one that we defined in the Codefresh Account Setting screen). That’s it! Now your build will automatically use this Docker Registry as a target. Click the Save button to apply the changes and then the Build button to start another build.
In the build logs a new panel will appear that shows the push progress:
Note that this is in addition to the internal Codefresh Docker storage. After the build is finished the Docker image of the sample application is stored both in the Codefresh internal registry and the public Docker Registry.
To verify the latter, you can visit your profile in Docker hub and look at the image details:
Pushing to the Docker Registry is the last step in the build pipeline. Now that we have the basic functionality ready we can see how Codefresh handles Continuous Integration with Pull requests and automatic builds.