Run your tests in parallel with Codefresh and Knapsack Pro

One of the most well-known problems when it comes to testing applications is the amount of time required by all test suites. Integration tests, in particular, are usually very slow to execute and depending on the type of application, several minutes (or even hours in extreme cases) are needed in order to get the final execution result.

You can reduce the test execution time with several techniques, but one of the most effective methods is running your tests in parallel. Knapsack Pro is a dedicated solution of test parallelization that can cut down the amount of test execution time by spreading test suites into multiple build nodes. Some of the most important features of Knapsack Pro are:

• Test parallelization for Ruby and Javascript applications (more languages coming)
• Support for popular test runners (rspec, cucumber, cypress, jest etc)
• Dynamic allocation of tests to build nodes (a.k.a. Queue mode)
• Support for short-lived build nodes (such as preemptible VMs on cloud providers)
• Fallback mode (run tests even when Knapsack API is not available)

It is very easy to use Knapsack Pro with Codefresh and split your tests in as many build nodes as you want:

In the example above we have used 5 parallel build executions to parallelize the test phase of the project.

Split your test executions in Codefresh pipelines

Codefresh already supports parallel pipeline steps out of the box. But we have recently added two new enhancements to the Codefresh YAML syntax that can make parallel pipelines even easier. These are the scale and matrix keywords.

The “scale” syntax can be easily used when you have multiple parallel steps that are mostly similar and only differ in one or more properties. The “matrix” syntax is the familiar way of creating matrix pipelines where you define all environment parameters and Codefresh will automatically create all possible combinations. For example, the following pipeline runs with 3 different versions of GO and 2 versions of the CGO switch.

version: '1.0'
stages:
  - prepare
  - test
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/cf-example-unit-test'
    revision: 'master'
    git: github
    stage: prepare
  run_my_tests_before_build:
    stage: test
    working_directory: './golang-app-A'
    commands:
     - go test -v
    matrix:
      image:
        - golang:1.11
        - golang:1.12
        - golang:1.13
      environment:
        - [CGO_ENABLED=1]
        - [CGO_ENABLED=0]     

The resulting pipeline will have 6 parallel steps for all the possible combinations.

Knapsack Pro can take advantage of this functionality by leveraging its API which only needs two parameters:

• KNAPSACK_PRO_CI_NODE_TOTAL
• KNAPSACK_PRO_CI_NODE_INDEX

The first parameter is needed only once in a pipeline and defines how many build nodes will be used to split tests. The second parameter should be declared for each node and contains a number that defines which node is that (0, 1, 2, 3 and so on).

By using these two parameters Knapsack Pro will automatically split your tests between nodes. There are two modes for the split. The Regular/Standard mode splits tests in a static manner by measuring how much time each test file takes by looking at the previous builds. The static set of tests is allocated only once to each parallel node before starting tests.

The dynamic/queue mode gives to each node only a subset of the tests and then monitors their execution within the same build before asking for another set of tests from the queue. A fast node (that finishes tests quickly) will then fetch more tests while a slow node will get fewer tests. This mode is great if the build nodes are not equal in resources or you have test files that sometimes take more or less time (often end to end tests can vary in time execution).

Codefresh supports both Knapsack Pro modes. Here is a full example that brings all this together:

version: "1.0"

stages:
  - "clone"
  - "build"
  - "tests"

steps:
  main_clone:
    type: "git-clone"
    description: "Cloning main repository..."
    repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}"
    revision: "${{CF_BRANCH}}"
    stage: "clone"
  BuildTestDockerImage:
    title: Building Test Docker image
    type: build
    arguments:
      image_name: "${{CF_ACCOUNT}}/${{CF_REPO_NAME}}-test"
      tag: "${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}"
      dockerfile: Test.Dockerfile
    stage: "build"

  run_tests:
    stage: "tests"
    image: "${{BuildTestDockerImage}}"
    working_directory: /src
    fail_fast: false
    environment:
      # set how many parallel jobs you want to run
      - KNAPSACK_PRO_CI_NODE_TOTAL=2
    matrix:
      environment:
        # please ensure you have here listed N-1 indexes
        # where N is KNAPSACK_PRO_CI_NODE_TOTAL
        - KNAPSACK_PRO_CI_NODE_INDEX=0
        - KNAPSACK_PRO_CI_NODE_INDEX=1
    commands:
      # run http server in the background (silent mode)
      # we did && echo on purpose to ensure Codefresh does not fail
      # when we pass npm process to background with & sign
      - (npm run start:ci &) && echo "start http server in the background"
      - $(npm bin)/knapsack-pro-cypress

This pipeline splits Cypress tests into two nodes:

Notice the values for KNAPSACK_PRO_CI_NODE_TOTAL and KNAPSACK_PRO_CI_NODE_INDEX. In order to run this pipeline in Codefresh, you also need a Knapsack Pro token that you can get after creating an account.

For more details see the integration page of Knapsack. Knapsack Pro tests also work great with service containers for running tests against databases or other external services. Here you can find articles dedicated to Knapsack Pro configuration for Ruby on Rails project on Codefresh and for Cypress end to end test runner.

Ready to try Codefresh, the CI/CD platform for Docker/Kubernetes/Helm? Create Your Free Account Today!