Fan-out-fan-in Pipeline
Use parallel mode to fan-in and fan-out your step dependencies
In pipelines, the concept of fan-in/fan-out is depicted in the diagram below. This pipeline offers parallel sub-flows in a single pipeline. Fan-out refers to spreading a task to multiple destinations in parallel, and fan-in is the opposite, where we spread multiple different tasks to the same destination.
As you can see by the diagram, Step1 fans out to Step2 and Step4 (which run in parallel), while Step3 and Step4 fan-in to Step5.
You can achieve parallelism in your Codefresh pipelines by using the following:
- Simple parallel jobs (inserting parallel steps into a sequential pipeline)
- Full parallel mode
- Fan-out/fan-in parallel pipelines, as covered here
Prerequisites
The Example Project
You can find the example Spring boot application on GitHub. It is a simple Hello World application that has several different types of tests we will be using to run using Codefresh’s parallel mode.
Create the Pipeline
Our pipeline will have five stages: setup, start, web-tests, smoke, and end:
You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you.
codefresh.yml
version: "1.0"
mode: parallel
stages:
- setup
- start
- web-tests
- smoke
- end
steps:
Clone:
title: Cloning main repository...
stage: setup
type: git-clone
arguments:
repo: codefresh-contrib/fan-out-fan-in-sample-app
git: github
revision: master
Build_image:
title: Building Docker Image...
type: build
stage: setup
working_directory: ${{Clone}}
arguments:
image_name: spring-backend
tag: latest
dockerfile: Dockerfile
when:
steps:
- name: Clone
on:
- success
Step1:
title: Running unit tests...
stage: start
working_directory: ${{Clone}}/complete
arguments:
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dgroups="unit" test
when:
steps:
- name: Build_image
on:
- success
services:
composition:
spring_backend:
image: ${{Build_image}}
ports:
- 8080
Step2:
title: Running web mock test...
stage: web-tests
working_directory: ${{Clone}}/complete
arguments:
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dgroups="web-mock" test
when:
steps:
- name: Step1
on:
- success
services:
composition:
spring_backend:
image: ${{Build_image}}
ports:
- 8080
Step3:
title: Running smoke test...
stage: smoke
working_directory: ${{Clone}}/complete
arguments:
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dgroups="smoke" test
when:
steps:
- name: Step2
on:
- success
services:
composition:
spring_backend:
image: ${{Build_image}}
ports:
- 8080
Step4:
title: Running web layer tests...
stage: web-tests
working_directory: ${{Clone}}/complete
arguments:
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dgroups="web-layer" test
when:
steps:
- name: Step1
on:
- success
services:
composition:
spring_backend:
image: ${{Build_image}}
ports:
- 8080
Step5:
title: Running integration tests...
stage: end
working_directory: ${{Clone}}/complete
arguments:
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dgroups="integration" test
when:
steps:
- name: Step3
on:
- success
- name: Step4
on:
- success
services:
composition:
spring_backend:
image: ${{Build_image}}
ports:
- 8080
Note the special use of mode: parallel
declared at the root of our yaml. This syntax makes the pipeline use the full parallel mode.
The order of your build steps doesn’t matter in this case, each step is executed according to its conditionals.
- Step1 (unit tests) fans out to Step2 and Step4 (web tests), which run in parallel
- Step3 (smoke tests) does not execute until Step2 is completed
- Step3 and Step4 fan in to the final step, Step5 (integration tests)
This pipeline consists of the following:
- A git-clone step that clones the main repository
- A build step that builds the cloned source code into a Docker image
- 5 freestyle steps that:
- Runs unit tests according to their respective @Tags
- Uses the image built in the second step as a Service container