Use parallel steps in your Codefresh pipelines

Use parallel steps in your Codefresh pipelines

6 min read

This is it! The moment you have been waiting for… The number one requested feature from several Codefresh customers is now in production. You can finally run parallel steps within Codefresh pipelines.

The reason this feature took so long to implement was that we wanted to offer the most flexible option possible. So after some lengthy internal testing, we are finally ready to present you not just one, but two different ways of running parallel pipeline steps:

  1. Injection of one or more parallel phases into a serial pipeline
  2. Full parallel mode – create your own pipeline graph

Let’s see these in order:

Basic parallel mode for CI/CD pipelines

The basic parallel mode is the easiest mode to use in Codefresh pipelines. It allows you to take any existing step in a serial pipeline and “split” into one or more parallel steps. Usually, the parallel steps will be of the same type (but this is not required). Here is an example where a Docker image is pushed in parallel in 3 Docker registries:

Docker parallel push
Docker parallel push

Here is the definition of this pipeline:

version: '1.0'
stages:
- build
- push
steps:
  MyAppDockerImage:
    title: Building Docker Image
    stage: 'build'
    type: build
    image_name: trivialgoweb
    working_directory: ./
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}'
    dockerfile: Dockerfile
  PushingToRegistries:
    type: parallel
    stage: 'push'
    steps:
      jfrog_PushingTo_jfrog_BintrayRegistry:
        type: push
        title: jfrog_Pushing To Bintray Registry
        candidate: ${{MyAppDockerImage}}
        tag: '${{CF_SHORT_REVISION}}'
        registry: bintray 
      PushingToGoogleRegistry:
        type: push
        title: Pushing To Google Registry
        candidate: ${{MyAppDockerImage}}
        tag: '${{CF_SHORT_REVISION}}'
        registry: gcr
      PushingToDockerRegistry:
        type: push
        title: Pushing To Dockerhub Registry
        candidate: ${{MyAppDockerImage}}
        tag: '${{CF_SHORT_REVISION}}'
        image_name: kkapelon/trivialgoweb
        registry: dockerhub 

Notice the type:parallel segment in the pipeline.

Unlike other solutions which allow you to use parallelism only in a specific pipeline segment, in Codefresh you can create parallelism with ANY type of step in a pipeline. Do you want parallel quality scanning? You have it. Interested in parallel deployments? You can do it. Linting while load testing? Anything is possible.

You can even use multiple parallel phases in a single pipeline. Here is an example where parallelism is used for both tests as well as deployments:

Parallel CI/CD pipeline
Parallel CI/CD pipeline

And here is the respective codefresh.yml:

version: '1.0'
stages:
- build
- 'parallel test'
- 'gcr push'
- 'parallel deploy'
steps:
  MyAppDockerImage:
    title: Building Docker Image
    stage: 'build'
    type: build
    image_name: my-app
    dockerfile: Dockerfile
  MyTestingPhase:
    type: parallel
    stage: 'parallel test'
    steps:
      myUnitTests:
        title: Unit testing
        image: ${{MyAppDockerImage}}
        commands:
        - echo "Tests"
      myIntegrationTests:
        title: Integration Testing
        image: ${{MyAppDockerImage}}
        commands:
        - echo "Integration"
  PushingToGoogleRegistry:
    type: push
    title: Pushing To GCR
    stage: 'gcr push'
    candidate: ${{MyAppDockerImage}}
    tag: '${{CF_SHORT_REVISION}}'
    registry: gcr     
  MyDeploymentPhase:
    type: parallel
    stage: 'parallel deploy'
    steps:
      deploy_to_env1:
        title: deployment ENV1
        type: deploy
        kind: kubernetes
        cluster:  myDemoCluster
        namespace: demo
        service: my-test-service
      deploy_to_env2:
        title: deployment ENV2
        type: deploy
        kind: kubernetes
        cluster:  myOtherDemoCluster
        namespace: demo
        service: my-test-service  

The pipeline is running both integration/load testing in parallel and then deploys to two environments at the same time.

Advanced parallel CI/CD pipeline mode

The basic parallel mode will cover most of the typical parallelism scenarios in Codefresh pipelines. We realized, however, that several organizations might want more flexibility on the types of parallelism needed in CI/CD pipelines.

A common example is parallel sub-flows in a single pipeline:

Parallel subflows in a CI/CD pipeline
Parallel subflows in a CI/CD pipeline

Instead of trying to predict every possible scenario needed by a company, we decided to offer an advanced parallel mode that can cater to all possible pipeline configurations. To use it you need to start your pipeline with:

version: '1.0'
mode: parallel
steps:
[...]

This syntax makes the pipeline use the full parallel mode. In this mode, the order of build steps in the codefresh.yml no longer matters. Instead, each step will be executed according to its dependencies. You have to specify the full order of steps using conditionals.

second_step:
  title: Second step
  when:
    steps:
     - name: first_step
       on:
         - success

In this example, we specify the step named “second_step” will only execute after “first_step” is finished.

Codefresh will automatically calculate the step order and create the critical path of the pipeline automatically. Once you run the pipeline, it will automatically make sure that each step runs in the correct order according to its dependencies. Here is the codefresh.yml that executes the pipeline shown in the previous picture:

version: '1.0'
mode: parallel
stages:
- start
- middleA
- middleB
- end

steps:
  Step1:
    title: Step1
    stage: start
    image: alpine
    commands:
      - echo "Step 1"
  Step2:
    title: Step2
    stage: middleA
    image: alpine
    commands:
      - echo "Step 2"
    when:
      steps:
       - name: Step1
         on:
           - success  
  Step3:
    title: Step3
    stage: middleB
    image: alpine
    commands:
      - echo "Step 3"
    when:
      steps:
       - name: Step2
         on:
           - success            
  Step4:
    title: Step4
    stage: middleB
    image: alpine
    commands:
      - echo "Step 4"
    when:
      steps:
       - name: Step1
         on:
           - success
  Step5:
    title: Step5
    stage: end
    image: alpine
    commands:
      - echo "Step 5"
    when:
      steps:
       - name: Step3
         on:
           - success
       - name: Step4
         on:
           - success   

The example CI/CD pipeline only uses dummy freestyle steps, so feel free to use it as a starting point with your own steps.

Here is also a sample representation (remember that pipeline stages are just used for visualization purposes and they don’t affect how a CI/CD pipeline is running)

Advanced parallel CI/CD pipelines
Advanced parallel CI/CD pipelines

Create your own pipeline graph

The advanced parallel mode described in the previous section is very powerful and we believe that it can cover a wide range of parallel scenarios for pipelines.

But we still decided to take it to the next level by giving you the full power of pipeline graphs. You can have as many “starting” steps as you want and as many “ending” steps. Your pipeline does not need to start and end in a single step.

This means that with Codefresh you can create fan-in-fan-out configurations like this:

Complex CI/CD pipeline graph
Complex CI/CD pipeline graph

This is the codefresh.yml file:

version: '1.0'
mode: parallel
stages:
- start
- middleA
- middleB
- middleC
- end

steps:
  Step1:
    title: Step1
    stage: start
    image: alpine
    commands:
      - echo "Step 1"
  Step2:
    title: Step2
    stage: middleA
    image: alpine
    commands:
      - echo "Step 2"
    when:
      steps:
       - name: Step1
         on:
           - success  
  Step3:
    title: Step3
    stage: middleB
    image: alpine
    commands:
      - echo "Step 3"
    when:
      steps:
       - name: Step2
         on:
           - success            
  Step4:
    title: Step4
    stage: middleB
    image: alpine
    commands:
      - echo "Step 4"
    when:
      steps:
       - name: Step1
         on:
           - success
  Step5:
    title: Step5
    stage: end
    image: alpine
    commands:
      - echo "Step 5"
    when:
      steps:
       - name: Step6
         on:
           - success
  Step6:
    title: Step6
    stage: middleC
    image: alpine
    commands:
      - echo "Step 6"
    when:
      steps:
       - name: Step4
         on:
           - success     
       - name: Step3
         on:
           - success             
  Step7:
    title: Step7
    stage: start
    image: alpine
    commands:
      - echo "Step 7"
  Step8:
    title: Step8
    stage: middleA
    image: alpine
    commands:
      - echo "Step 8"
    when:
      steps:
       - name: Step7
         on:
           - success
  Step9:
    title: Step9
    stage: middleC
    image: alpine
    commands:
      - echo "Step 9"
    when:
      steps:
       - name: Step4
         on:
           - success
       - name: Step8
         on:
           - success
  Step10:
    title: Step10
    stage: end
    image: alpine
    commands:
      - echo "Step 10"
    when:
      steps:
       - name: Step6
         on:
           - success
           

The possibilities are endless. The parallel mode has even more features not mentioned in this blog post. See the full documentation page for more details.

Ready to try Codefresh, the CI/CD platform for Docker/Kubernetes/Helm? Create Your Free Account Today!

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments