Create your FREE Codefresh account and start making pipelines fast. Create Account

Use parallel steps in your Codefresh pipelines

4 min read

This is it! The moment you have been waiting for… The number one requested feature from several Codefresh customers is now in production. You can finally run parallel steps within Codefresh pipelines.

The reason this feature took so long to implement was that we wanted to offer the most flexible option possible. So after some lengthy internal testing, we are finally ready to present you not just one, but two different ways of running parallel pipeline steps:

  1. Injection of one or more parallel phases into a serial pipeline
  2. Full parallel mode – create your own pipeline graph

Let’s see these in order:

Basic parallel mode for CI/CD pipelines

The basic parallel mode is the easiest mode to use in Codefresh pipelines. It allows you to take any existing step in a serial pipeline and “split” into one or more parallel steps. Usually, the parallel steps will be of the same type (but this is not required). Here is an example where a Docker image is pushed in parallel in 3 Docker registries:

Docker parallel push
Docker parallel push

Here is the definition of this pipeline:

Notice the type:parallel segment in the pipeline.

Unlike other solutions which allow you to use parallelism only in a specific pipeline segment, in Codefresh you can create parallelism with ANY type of step in a pipeline. Do you want parallel quality scanning? You have it. Interested in parallel deployments? You can do it. Linting while load testing? Anything is possible.

You can even use multiple parallel phases in a single pipeline. Here is an example where parallelism is used for both tests as well as deployments:

Parallel CI/CD pipeline
Parallel CI/CD pipeline

And here is the respective codefresh.yml:

The pipeline is running both integration/load testing in parallel and then deploys to two environments at the same time.

Advanced parallel CI/CD pipeline mode

The basic parallel mode will cover most of the typical parallelism scenarios in Codefresh pipelines. We realized, however, that several organizations might want more flexibility on the types of parallelism needed in CI/CD pipelines.

A common example is parallel sub-flows in a single pipeline:

Parallel subflows in a CI/CD pipeline
Parallel subflows in a CI/CD pipeline

Instead of trying to predict every possible scenario needed by a company, we decided to offer an advanced parallel mode that can cater to all possible pipeline configurations. To use it you need to start your pipeline with:

This syntax makes the pipeline use the full parallel mode. In this mode, the order of build steps in the codefresh.yml no longer matters. Instead, each step will be executed according to its dependencies. You have to specify the full order of steps using conditionals.

In this example, we specify the step named “second_step” will only execute after “first_step” is finished.

Codefresh will automatically calculate the step order and create the critical path of the pipeline automatically. Once you run the pipeline, it will automatically make sure that each step runs in the correct order according to its dependencies. Here is the codefresh.yml that executes the pipeline shown in the previous picture:

The example CI/CD pipeline only uses dummy freestyle steps, so feel free to use it as a starting point with your own steps.

Here is also a sample representation (remember that pipeline stages are just used for visualization purposes and they don’t affect how a CI/CD pipeline is running)

Advanced parallel CI/CD pipelines
Advanced parallel CI/CD pipelines

Create your own pipeline graph

The advanced parallel mode described in the previous section is very powerful and we believe that it can cover a wide range of parallel scenarios for pipelines.

But we still decided to take it to the next level by giving you the full power of pipeline graphs. You can have as many “starting” steps as you want and as many “ending” steps. Your pipeline does not need to start and end in a single step.

This means that with Codefresh you can create fan-in-fan-out configurations like this:

Complex CI/CD pipeline graph
Complex CI/CD pipeline graph

This is the codefresh.yml file:

The possibilities are endless. The parallel mode has even more features not mentioned in this blog post. See the full documentation page for more details.

Ready to try Codefresh, the CI/CD platform for Docker/Kubernetes/Helm? Create Your Free Account Today!

Kostis Kapelonis

About Kostis Kapelonis

Kostis is a software engineer/technical-writer dual class character. He lives and breathes automation, good testing practices and stress-free deployments.

Enjoy this article? Don't forget to share.

Comments

* All fields are required. Your email address will not be published.

Related Posts

Get a head start on building better pipelines!

Schedule a demo with a Codefresh expert today.