Unit Testing

How to run Unit tests in Codefresh pipelines

With Codefresh, you can easily run unit tests for every commit or pull request.

For the purposes of this guide, “unit tests” are the tests that use only the source code of the application and nothing else. If you are interested in running tests with external services (such as databases) then see the page about integration tests.

Different companies have different types of unit tests and in several cases the type of programming language also affects when/what tests are run. Codefresh supports all test frameworks (including mocking frameworks) for all popular programming languages.

In this page we will see 4 ways of running unit tests in Codefresh:

1. Running unit tests in a dockerfile (recommended only for smoke tests)
2. Running unit tests with an external image (best for traditional/simple applications)
3. Running unit tests in the application image (not recommended, but very popular)
4. Running unit tests using a special testing image (the recommended solution for complex applications)

For an example application of the second and third ways see the unit test example page

Running Unit tests as part of a Docker build

A handy way to quickly test a Docker image, is by placing one or more smoke tests in the Dockerfile itself. The unit tests then execute when the image is built, and if they fail the image is not even created. Here is an example:

Dockerfile

FROM python:3.6.4-alpine3.6

ENV FLASK_APP=minitwit
COPY . /app
WORKDIR /app

RUN pip install --editable .
RUN flask initdb

# Unit tests
RUN python setup.py test

EXPOSE 5000
CMD [ "flask", "run", "--host=0.0.0.0" ]

This kind of unit tests are transparent to Codefresh. They will just execute in a build step in the same manner as if you would build the image on your workstation.

A big disadvantage of this unit testing method, is that getting reports out of the docker image is not a straightforward process. On the other hand they are very easy to integrate in any workflow. The Codefresh pipeline simply checks out the code and builds a Dockerfile.

codefresh.yml

version: '1.0'
stages:
  - prepare
  - build
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefresh-contrib/python-flask-sample-app'
    revision: 'with-tests-in-dockerfile'
    stage: prepare
    git: github
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: build
    image_name: my-app-image
    working_directory: ./
    tag: with-tests-in-dockerfile
    dockerfile: Dockerfile

This technique is best used for a very small subset of unit tests that check the overall well being of a Docker image. The bulk of the tests should be executed outside the Docker build process as we will see in the next sections.

Running Unit tests using an external Docker image

The recommended way to run unit tests in Codefresh pipelines is to select a Docker image that has all the test tools that you need and define an explicit testing step in your pipeline (usually a freestyle step).

Here is an example where a JDK/Maven image is used to run unit tests:

codefresh.yml

version: '1.0'
stages:
  - prepare
  - test
  - package
steps:
  main_clone:
    title: Cloning main repository...
    stage: prepare
    type: git-clone
    repo: 'codefresh-contrib/spring-boot-2-sample-app'
    revision: master
    git: github
  MyUnitTests:
    title: JUnit tests
    stage: test
    image: 'maven:3.5.2-jdk-8-alpine'
    commands:
      - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository test
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: package
    image_name: spring-boot-2-sample-app
    working_directory: ./
    tag: 'non-multi-stage'
    dockerfile: Dockerfile     

The big advantage of this approach is that you can easily replicate your test environment in the Codefresh pipeline by selecting the appropriate image for your tests. You also get a clear overview on the test results. If they fail, the pipeline will automatically stop. You can change this behavior with the fail_fast property.

Notice that even if the example above, creates eventually a Docker image, you can still use this way of running unit tests for traditional applications that are not dockerized yet (e.g. VM-based applications).

Running Unit tests with the Application Docker image

In several cases (especially with dynamic languages) the Docker image that holds the application can also be re-used for unit tests. This is a very common technique for Node, Python and Ruby applications. In this case you can use the context feature of Codefresh to run a unit test step in the image that was created in a previous step:

codefresh.yml

version: '1.0'
stages:
  - prepare
  - build
  - test
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefresh-contrib/python-flask-sample-app'
    revision: 'master'
    git: github
    stage: prepare
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: build
    image_name: my-app-image
    working_directory: ./
    tag: 'master'
    dockerfile: Dockerfile
  MyUnitTests:
    title: Running Unit tests
    stage: test
    image: '${{MyAppDockerImage}}'
    commands:
      - python setup.py test     

Notice that here we use a Codefresh variable as the value of the image property in the last step. This will make the unit test execute in the same docker container that was created in the second step of the pipeline.

This technique is certainly useful, but can be easily abused if you end up shipping testing tools in your production image (which is not recommended). If you find your production images filled with test tools and libraries it is better to use the technique in the next section which uses a different image for tests.

Running Unit tests with a dynamic Docker image

The ultimate way of running unit tests in Codefresh is by creating a specific image dedicated to unit tests. If Dockerhub doesn’t already contain an image that suits you, you should instead create your own.

This means that your application has two Dockerfiles. The main one that holds the application as a deployment artifact and a separate one that holds all the unit test libraries and tools that you need. Here is an example:

codefresh.yml

version: '1.0'
stages:
  - prepare
  - 'Test tools'
  - test
  - build
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/demochat'
    revision: 'master'
    git: github
    stage: prepare
  MyUnitTestDockerImage:
    title: Building Test image
    type: build
    stage: 'Test tools'
    image_name: my-test-image
    working_directory: ./
    tag: 'master'
    dockerfile: Dockerfile.dev
  MyUnitTests:
    title: Running Unit tests
    stage: test
    image: '${{MyUnitTestDockerImage}}'
    commands:
      - npm run test
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    stage: build
    image_name: my-app-image
    working_directory: ./
    tag: 'master'
    dockerfile: Dockerfile    

Notice that here we create two Docker images.

1. The first docker image is created from Dockerfile.dev
2. Unit tests run in the context of that image (MyUnitTestDockerImage)
3. The production application uses another Dockerfile

This is one of the best ways to run unit tests (as well as integration tests) as it allows you to fine tune the test environment while still shipping to production only what is needed.

In the example above we used two different dockerfiles, but you could also use a single dockerfile with multi-stage builds and use the target directive to stop the image build process at a previous layer (that has all the testing tools).

Creating Test reports

All the ways mentioned above for running unit tests (apart of the first one), can also be used for test reporting.

Read all about test results and graphs in the test reports page.

What to read next

• Unit test example
• Introduction to Pipelines
• Codefresh YAML
• On demand environments
• Integration tests