This is it! The moment you have been waiting for… The number one requested feature from several Codefresh customers is now in production. You can finally run parallel steps within Codefresh pipelines.
The reason this feature took so long to implement was that we wanted to offer the most flexible option possible. So after some lengthy internal testing, we are finally ready to present you not just one, but two different ways of running parallel pipeline steps:
- Injection of one or more parallel phases into a serial pipeline
- Full parallel mode – create your own pipeline graph
Let’s see these in order:
Basic parallel mode for CI/CD pipelines
The basic parallel mode is the easiest mode to use in Codefresh pipelines. It allows you to take any existing step in a serial pipeline and “split” into one or more parallel steps. Usually, the parallel steps will be of the same type (but this is not required). Here is an example where a Docker image is pushed in parallel in 3 Docker registries:
Here is the definition of this pipeline:
version: '1.0'
stages:
- build
- push
steps:
MyAppDockerImage:
title: Building Docker Image
stage: 'build'
type: build
image_name: trivialgoweb
working_directory: ./
tag: '${{CF_BRANCH_TAG_NORMALIZED}}'
dockerfile: Dockerfile
PushingToRegistries:
type: parallel
stage: 'push'
steps:
jfrog_PushingTo_jfrog_BintrayRegistry:
type: push
title: jfrog_Pushing To Bintray Registry
candidate: ${{MyAppDockerImage}}
tag: '${{CF_SHORT_REVISION}}'
registry: bintray
PushingToGoogleRegistry:
type: push
title: Pushing To Google Registry
candidate: ${{MyAppDockerImage}}
tag: '${{CF_SHORT_REVISION}}'
registry: gcr
PushingToDockerRegistry:
type: push
title: Pushing To Dockerhub Registry
candidate: ${{MyAppDockerImage}}
tag: '${{CF_SHORT_REVISION}}'
image_name: kkapelon/trivialgoweb
registry: dockerhub
Notice the type:parallel segment in the pipeline.
Unlike other solutions which allow you to use parallelism only in a specific pipeline segment, in Codefresh you can create parallelism with ANY type of step in a pipeline. Do you want parallel quality scanning? You have it. Interested in parallel deployments? You can do it. Linting while load testing? Anything is possible.
You can even use multiple parallel phases in a single pipeline. Here is an example where parallelism is used for both tests as well as deployments:
And here is the respective codefresh.yml:
version: '1.0'
stages:
- build
- 'parallel test'
- 'gcr push'
- 'parallel deploy'
steps:
MyAppDockerImage:
title: Building Docker Image
stage: 'build'
type: build
image_name: my-app
dockerfile: Dockerfile
MyTestingPhase:
type: parallel
stage: 'parallel test'
steps:
myUnitTests:
title: Unit testing
image: ${{MyAppDockerImage}}
commands:
- echo "Tests"
myIntegrationTests:
title: Integration Testing
image: ${{MyAppDockerImage}}
commands:
- echo "Integration"
PushingToGoogleRegistry:
type: push
title: Pushing To GCR
stage: 'gcr push'
candidate: ${{MyAppDockerImage}}
tag: '${{CF_SHORT_REVISION}}'
registry: gcr
MyDeploymentPhase:
type: parallel
stage: 'parallel deploy'
steps:
deploy_to_env1:
title: deployment ENV1
type: deploy
kind: kubernetes
cluster: myDemoCluster
namespace: demo
service: my-test-service
deploy_to_env2:
title: deployment ENV2
type: deploy
kind: kubernetes
cluster: myOtherDemoCluster
namespace: demo
service: my-test-service
The pipeline is running both integration/load testing in parallel and then deploys to two environments at the same time.
Advanced parallel CI/CD pipeline mode
The basic parallel mode will cover most of the typical parallelism scenarios in Codefresh pipelines. We realized, however, that several organizations might want more flexibility on the types of parallelism needed in CI/CD pipelines.
A common example is parallel sub-flows in a single pipeline:
Instead of trying to predict every possible scenario needed by a company, we decided to offer an advanced parallel mode that can cater to all possible pipeline configurations. To use it you need to start your pipeline with:
version: '1.0' mode: parallel steps: [...]
This syntax makes the pipeline use the full parallel mode. In this mode, the order of build steps in the codefresh.yml no longer matters. Instead, each step will be executed according to its dependencies. You have to specify the full order of steps using conditionals.
second_step:
title: Second step
when:
steps:
- name: first_step
on:
- success
In this example, we specify the step named “second_step” will only execute after “first_step” is finished.
Codefresh will automatically calculate the step order and create the critical path of the pipeline automatically. Once you run the pipeline, it will automatically make sure that each step runs in the correct order according to its dependencies. Here is the codefresh.yml that executes the pipeline shown in the previous picture:
version: '1.0'
mode: parallel
stages:
- start
- middleA
- middleB
- end
steps:
Step1:
title: Step1
stage: start
image: alpine
commands:
- echo "Step 1"
Step2:
title: Step2
stage: middleA
image: alpine
commands:
- echo "Step 2"
when:
steps:
- name: Step1
on:
- success
Step3:
title: Step3
stage: middleB
image: alpine
commands:
- echo "Step 3"
when:
steps:
- name: Step2
on:
- success
Step4:
title: Step4
stage: middleB
image: alpine
commands:
- echo "Step 4"
when:
steps:
- name: Step1
on:
- success
Step5:
title: Step5
stage: end
image: alpine
commands:
- echo "Step 5"
when:
steps:
- name: Step3
on:
- success
- name: Step4
on:
- success
The example CI/CD pipeline only uses dummy freestyle steps, so feel free to use it as a starting point with your own steps.
Here is also a sample representation (remember that pipeline stages are just used for visualization purposes and they don’t affect how a CI/CD pipeline is running)
Create your own pipeline graph
The advanced parallel mode described in the previous section is very powerful and we believe that it can cover a wide range of parallel scenarios for pipelines.
But we still decided to take it to the next level by giving you the full power of pipeline graphs. You can have as many “starting” steps as you want and as many “ending” steps. Your pipeline does not need to start and end in a single step.
This means that with Codefresh you can create fan-in-fan-out configurations like this:
This is the codefresh.yml file:
version: '1.0'
mode: parallel
stages:
- start
- middleA
- middleB
- middleC
- end
steps:
Step1:
title: Step1
stage: start
image: alpine
commands:
- echo "Step 1"
Step2:
title: Step2
stage: middleA
image: alpine
commands:
- echo "Step 2"
when:
steps:
- name: Step1
on:
- success
Step3:
title: Step3
stage: middleB
image: alpine
commands:
- echo "Step 3"
when:
steps:
- name: Step2
on:
- success
Step4:
title: Step4
stage: middleB
image: alpine
commands:
- echo "Step 4"
when:
steps:
- name: Step1
on:
- success
Step5:
title: Step5
stage: end
image: alpine
commands:
- echo "Step 5"
when:
steps:
- name: Step6
on:
- success
Step6:
title: Step6
stage: middleC
image: alpine
commands:
- echo "Step 6"
when:
steps:
- name: Step4
on:
- success
- name: Step3
on:
- success
Step7:
title: Step7
stage: start
image: alpine
commands:
- echo "Step 7"
Step8:
title: Step8
stage: middleA
image: alpine
commands:
- echo "Step 8"
when:
steps:
- name: Step7
on:
- success
Step9:
title: Step9
stage: middleC
image: alpine
commands:
- echo "Step 9"
when:
steps:
- name: Step4
on:
- success
- name: Step8
on:
- success
Step10:
title: Step10
stage: end
image: alpine
commands:
- echo "Step 10"
when:
steps:
- name: Step6
on:
- success
The possibilities are endless. The parallel mode has even more features not mentioned in this blog post. See the full documentation page for more details.
Ready to try Codefresh, the CI/CD platform for Docker/Kubernetes/Helm? Create Your Free Account Today!