This is it! The moment you have been waiting for… The number one requested feature from several Codefresh customers is now in production. You can finally run parallel steps within Codefresh pipelines.
The reason this feature took so long to implement was that we wanted to offer the most flexible option possible. So after some lengthy internal testing, we are finally ready to present you not just one, but two different ways of running parallel pipeline steps:
- Injection of one or more parallel phases into a serial pipeline
- Full parallel mode – create your own pipeline graph
Let’s see these in order:
Basic parallel mode for CI/CD pipelines
The basic parallel mode is the easiest mode to use in Codefresh pipelines. It allows you to take any existing step in a serial pipeline and “split” into one or more parallel steps. Usually, the parallel steps will be of the same type (but this is not required). Here is an example where a Docker image is pushed in parallel in 3 Docker registries:
Here is the definition of this pipeline:
version: '1.0' stages: - build - push steps: MyAppDockerImage: title: Building Docker Image stage: 'build' type: build image_name: trivialgoweb working_directory: ./ tag: '${{CF_BRANCH_TAG_NORMALIZED}}' dockerfile: Dockerfile PushingToRegistries: type: parallel stage: 'push' steps: jfrog_PushingTo_jfrog_BintrayRegistry: type: push title: jfrog_Pushing To Bintray Registry candidate: ${{MyAppDockerImage}} tag: '${{CF_SHORT_REVISION}}' registry: bintray PushingToGoogleRegistry: type: push title: Pushing To Google Registry candidate: ${{MyAppDockerImage}} tag: '${{CF_SHORT_REVISION}}' registry: gcr PushingToDockerRegistry: type: push title: Pushing To Dockerhub Registry candidate: ${{MyAppDockerImage}} tag: '${{CF_SHORT_REVISION}}' image_name: kkapelon/trivialgoweb registry: dockerhub
Notice the type:parallel
segment in the pipeline.
Unlike other solutions which allow you to use parallelism only in a specific pipeline segment, in Codefresh you can create parallelism with ANY type of step in a pipeline. Do you want parallel quality scanning? You have it. Interested in parallel deployments? You can do it. Linting while load testing? Anything is possible.
You can even use multiple parallel phases in a single pipeline. Here is an example where parallelism is used for both tests as well as deployments:
And here is the respective codefresh.yml:
version: '1.0' stages: - build - 'parallel test' - 'gcr push' - 'parallel deploy' steps: MyAppDockerImage: title: Building Docker Image stage: 'build' type: build image_name: my-app dockerfile: Dockerfile MyTestingPhase: type: parallel stage: 'parallel test' steps: myUnitTests: title: Unit testing image: ${{MyAppDockerImage}} commands: - echo "Tests" myIntegrationTests: title: Integration Testing image: ${{MyAppDockerImage}} commands: - echo "Integration" PushingToGoogleRegistry: type: push title: Pushing To GCR stage: 'gcr push' candidate: ${{MyAppDockerImage}} tag: '${{CF_SHORT_REVISION}}' registry: gcr MyDeploymentPhase: type: parallel stage: 'parallel deploy' steps: deploy_to_env1: title: deployment ENV1 type: deploy kind: kubernetes cluster: myDemoCluster namespace: demo service: my-test-service deploy_to_env2: title: deployment ENV2 type: deploy kind: kubernetes cluster: myOtherDemoCluster namespace: demo service: my-test-service
The pipeline is running both integration/load testing in parallel and then deploys to two environments at the same time.
Advanced parallel CI/CD pipeline mode
The basic parallel mode will cover most of the typical parallelism scenarios in Codefresh pipelines. We realized, however, that several organizations might want more flexibility on the types of parallelism needed in CI/CD pipelines.
A common example is parallel sub-flows in a single pipeline:
Instead of trying to predict every possible scenario needed by a company, we decided to offer an advanced parallel mode that can cater to all possible pipeline configurations. To use it you need to start your pipeline with:
version: '1.0' mode: parallel steps: [...]
This syntax makes the pipeline use the full parallel mode. In this mode, the order of build steps in the codefresh.yml no longer matters. Instead, each step will be executed according to its dependencies. You have to specify the full order of steps using conditionals.
second_step: title: Second step when: steps: - name: first_step on: - success
In this example, we specify the step named “second_step” will only execute after “first_step” is finished.
Codefresh will automatically calculate the step order and create the critical path of the pipeline automatically. Once you run the pipeline, it will automatically make sure that each step runs in the correct order according to its dependencies. Here is the codefresh.yml that executes the pipeline shown in the previous picture:
version: '1.0' mode: parallel stages: - start - middleA - middleB - end steps: Step1: title: Step1 stage: start image: alpine commands: - echo "Step 1" Step2: title: Step2 stage: middleA image: alpine commands: - echo "Step 2" when: steps: - name: Step1 on: - success Step3: title: Step3 stage: middleB image: alpine commands: - echo "Step 3" when: steps: - name: Step2 on: - success Step4: title: Step4 stage: middleB image: alpine commands: - echo "Step 4" when: steps: - name: Step1 on: - success Step5: title: Step5 stage: end image: alpine commands: - echo "Step 5" when: steps: - name: Step3 on: - success - name: Step4 on: - success
The example CI/CD pipeline only uses dummy freestyle steps, so feel free to use it as a starting point with your own steps.
Here is also a sample representation (remember that pipeline stages are just used for visualization purposes and they don’t affect how a CI/CD pipeline is running)
Create your own pipeline graph
The advanced parallel mode described in the previous section is very powerful and we believe that it can cover a wide range of parallel scenarios for pipelines.
But we still decided to take it to the next level by giving you the full power of pipeline graphs. You can have as many “starting” steps as you want and as many “ending” steps. Your pipeline does not need to start and end in a single step.
This means that with Codefresh you can create fan-in-fan-out configurations like this:
This is the codefresh.yml file:
version: '1.0' mode: parallel stages: - start - middleA - middleB - middleC - end steps: Step1: title: Step1 stage: start image: alpine commands: - echo "Step 1" Step2: title: Step2 stage: middleA image: alpine commands: - echo "Step 2" when: steps: - name: Step1 on: - success Step3: title: Step3 stage: middleB image: alpine commands: - echo "Step 3" when: steps: - name: Step2 on: - success Step4: title: Step4 stage: middleB image: alpine commands: - echo "Step 4" when: steps: - name: Step1 on: - success Step5: title: Step5 stage: end image: alpine commands: - echo "Step 5" when: steps: - name: Step6 on: - success Step6: title: Step6 stage: middleC image: alpine commands: - echo "Step 6" when: steps: - name: Step4 on: - success - name: Step3 on: - success Step7: title: Step7 stage: start image: alpine commands: - echo "Step 7" Step8: title: Step8 stage: middleA image: alpine commands: - echo "Step 8" when: steps: - name: Step7 on: - success Step9: title: Step9 stage: middleC image: alpine commands: - echo "Step 9" when: steps: - name: Step4 on: - success - name: Step8 on: - success Step10: title: Step10 stage: end image: alpine commands: - echo "Step 10" when: steps: - name: Step6 on: - success
The possibilities are endless. The parallel mode has even more features not mentioned in this blog post. See the full documentation page for more details.
Ready to try Codefresh, the CI/CD platform for Docker/Kubernetes/Helm? Create Your Free Account Today!