How to define pipelines in Codefresh
Now that we know the theory behind Codefresh pipelines, we can see how you can create pipelines for your own project.
The main concepts are shown below:
Projects are the top level concept in Codefresh. You can create projects to group pipelines that are related. In most cases a single project will be a single application (that itself contains many micro-services). You are free to use projects as you see fit. For example you could create a project for a specific Kubernetes cluster or a specific team/department.
Each project can have multiple pipelines. Pipelines that belong to a single project are easily managed all together. It is also very easy to create a new pipeline in a project by copying an existing pipeline.
Each pipeline has a definition that defines the pipeline steps that are executed each time this pipeline is triggered. The definition of a pipeline is described in special codefresh.yml file. The
codefresh.ymlfile can be fetched from the same repository of the source code, from a completely different repository or even defined in-place in the Codefresh pipeline editor.
Each pipeline can have zero, one or more triggers. Codefresh supports several kind of triggers such as Git, Cron or Docker push triggers. Triggers that happen with Git webhooks can come from the same git repository that contains the git code OR any other completely different repository.
With these basic building blocks you can define many complex workflows.
Creating new pipelines
You can create new projects by clicking on Projects in the left sidebar and then selecting the Add Project button on the top right corner. A dialog will appear that will ask you for the project name and optional tags that you can use for access control.
Once you are inside the project view you can start editing pipelines with a UI environment that works similar to a traditional IDE.
On the top left you can see your current project. You can also change it by clicking on the drop-down on the top left corner.
On the left side of the screen you can see all pipelines that currently belong to this project. Click on each one to edit it. On the bottom part of this panel the Add pipeline button allows you to create a new pipeline on the same project either from scratch or by copying an existing one from the same project or a completely different project.
The name of the currently edited pipeline is shown at the top of the window
The main window shows the definition of the current pipeline. The screenshot shows the inline editor but pipeline can also be defined from external files (checked into source control) as explained later.
The right part of the window shows extra settings for this pipeline such as triggers and launch variables/parameters.
Using the inline pipeline editor
When first creating a pipeline you will see an inline editor that allows you to define the pipeline yml right there in the Codefresh UI. This is great when you are starting a new project because it offers you really quick feedback. You can edit the yml steps, run a build, edit again, run a build and so on.
On the top right of the panel you have additional controls
- The import button allows you to bring a
codefresh.ymlfrom your local workstation into the editor
- The comment button allows you to quickly comment/uncomment the currently selected text. The hotkey
Ctrl-/also performs the same action
- The formatting button enriches the editor with special symbols for linebreaks, spaces and tabs. This allows you to easily fix common formatting errors
- The copy button quickly copies the whole pipeline text in your clipboard.
Notice that in the editor you can expand/collapse individual yaml blocks using the arrow triangles on the left of each blocks
Loading codefresh.yml from version control
Working with the inline editor is very convenient in the beginning, but it makes your pipeline definition only exist with the Codefresh UI and therefore goes against the basic principles of infrastructure as code. Once you are happy with how your pipeline works you should commit it to your repository.
You can click on the Inline YAML header and switch it to Use YAML from repository.
You can then select any git repository accessible to you and load the
codefresh.yml from there. You also setup two additional settings:
- The path of the file inside the repository. This allows you to have special folders for pipeline definitions
- The branch of the repository to use for loading the
codefresh.ymlfile if you have more than one.
In the branch drop down you can also choose the option DYNAMIC. This will use the same branch as the one mentioned in the trigger event. If for example your pipeline is triggered by a commit in the
staging branch of the source code, the pipeline definition will also be loaded from the
staging branch of the git repository that contains the
This allows you to have complex pipeline definitions per branch but you have to be careful to match git branch names between the repository that holds the source code and the repository that holds the pipeline definition. It is much easier to pick a specific branch for the pipeline definitions that will always be the same.
It is also possible to switch the header to Use YAML from URL. This will allow you to load a codefresh yaml from any public URL. Notice that a raw URL is needed in the case of github. As an example instead of using
https://github.com/codefresh-contrib/example-voting-app/blob/master/codefresh.yml you should enter
Legacy Repository pipelines
If you have a Codefresh account created before May 2019 you will still get access to the Repository view. This view will still be available for a transition period to help you migrate to the project concept. Both views are still valid and operating on the same pipelines behind the scenes.
All you repositories should be migrated on projects with the same name so you can edit your pipelines from either view.
You can also add detached pipelines (i.e. pipelines not connected to a repository or project) manually from the
This procedure is only needed for detached pipelines which do not belong to a repository. Pipelines that were connected to a git repository will also be connected to the project with the same name.
Pipeline creation modes (Legacy)
You can start the creation of a pipeline from two places in the Codefresh UI
- From a specific repository as found in the repositories view
- From the dedicated pipelines view on the left sidebar
Both ways are equally valid and differ only in the way the created pipeline is accessing a git repository.
Creating a pipeline from a GIT repository gives you immediate access to the contents of that repository. When you run the pipeline, Codefresh will automatically checkout the contents of the GIT repository inside the workspace folder. Your pipeline can then focus on building and packaging the source code.
Creating a pipeline directly attached to a GIT repository is the recommended way to start using Codefresh pipelines that deal with source code. In most cases, this is the type of pipeline you will use when you start using Codefresh.
If you choose to create instead a pipeline from the Pipelines view from the left sidebar, the workspace will start completely empty and it is your responsibility to manually checkout code by using the git clone step.
This is a more advanced way of creating pipelines. It is ideal if you are creating pipelines that don’t deal strictly with source code (e.g. a pipeline that promotes artifacts between Docker repositories) or pipelines that work with multiple git repositories.
If you are unsure which way is appropriate for you, then choose the first one and create your pipelines starting from a GIT repository.
Regardless of the method you select, you will reach the same pipeline definition screen that allows you to define the individual build steps.
Pipeline definition modes (Legacy)
There are 3 ways to define the build steps of pipelines in Codefresh
- Using the GUI (easy but not very flexible)
- Using a Codefresh YML file (recommended)
- Programmatically using the API (Advanced)
We recommend you start with the GUI way if are you still learning about containers and deployments, but for any non trivial project you will soon discover that using the Codefresh YML file is the most flexible way.
Creating pipelines using the Codefresh GUI (Legacy)
Creating a pipeline via the GUI is the fastest way of getting a Docker image from your source code and optionally deploying into Kubernetes if your application matches the expectation of Codefresh.
The GUI steps in Codefresh assume that
- Your git repository contains a single application with a single Dockerfile
- The Dockerfile contains both the application as well as any libraries needed for unit testing
- The docker image that will be created will be tagged with the name of the branch that built it
- Your Kubernetes cluster either contains a deployed service already or you have a deployment manifest in the repository
If your application does not match these expectations, then you need to use a
codefresh.yml file instead (explained later in this page).
The predefined steps for the GUI are shown in the pipeline screen as different sections. You can expand any of these and change their settings.
The Build step creates a Docker image from your Dockerfile. Note that the Dockerfile should be self-contained
and compile/package all needed resources on its own. If you have a Dockerfile that expects something to be prepared in advance, you need to switch your build to use a
codefresh.yml file instead.
The Unit test step executes any command of your choosing inside the Docker image that was created in the previous step. This means that if you need any special unit test libraries, your Dockerfile should keep them intact (So if you are using multi-stage Docker builds, you cannot execute tests using the GUI way). If your unit tests fail, the pipeline stops and the Docker image is not uploaded in any Docker registry.
The Push to registry steps pushes the Docker image to any registry of your choosing as defined in external registry integrations. Notice that your image will be uploaded always to the integrated Codefresh Docker registry if you don’t do something special.
The Integrations tests step executes any command of your choosing inside the Docker image that was created in the Build step so the same caveats as unit tests apply. If the integration tests fail however, the Docker image is already uploaded in the image registry.
For both unit and integration tests you can use Codefresh compositions which allow you to launch your container with other external services (such as a database) and run a more complete environment. For more details see the compositions documentation.
The Deploy Script step contains some templates for deployment. You can deploy to Kubernetes, ECS and Docker swarm or run a custom deployment command on the container that was created in the build step or any other image of your choosing.
Creating pipelines using Codefresh YML (Legacy)
The Codefresh YML option offers a special syntax for defining your builds in ways that are not possible with the predefined pipeline steps. With the YML option you can
- create multiple Docker images
- customize the Docker tags
- upload to multiple Docker repositories
- control exactly the services that run during your unit/integration tests
- re-arrange the order of steps
- Use custom images for compilation/package steps
- and much more
To switch to the flexible yml configuration, click the YAML toggle on the top right of the interface
Once you switch to YML mode you have 3 more options on how to select the yml content
- Inline yml (This is for the initial creation of the project)
- Read yml from repository (recommended)
- Read yml from URL (allows re-use of yml files between different projects)
Writing Codefresh YML in the GUI
The inline option allows you to define the build yml right there in the Codefresh UI. This is great when you are starting a new project because it offers you really quick feedback. You can edit the yml steps, run a build, edit again, run a build and so on. Even though this is very convenient in the beginning, it makes your pipeline definition only exist with the Codefresh UI and therefore goes against the basic principles of infrastructure as code. Once you are happy with how your pipeline works you should commit it to your repository and use the second option.
You can also import directly yml steps from a file on your computer as a starting point by clicking the import from file button.
Notice that in the editor you can expand/collapse individual yaml blocks using the arrow triangles on the left.
Also you can comment/uncomment any block of code that you select either with the on-screen button or the
Ctrl-/ keyboard shortcut.
Using a codefresh.yml for the source code repository
The repository option is the recommended on. It reads the
codefresh.yml file from the repository that contains your source code. This way when you change the file you also get history and auditing for free via the GIT functionality. Both the name and location of the file are configurable.
This choice is only available if the pipeline was created from a git repository. It is not available for pipelines created from the dedicated Pipelines view of the left sidebar.
Share single Codefresh YAML across different pipelines
The third option allows you to load the yml from any location, even from a different repository. This allows you to create yml files in a central repository or web server and reuse them in multiple Codefresh pipelines. So if you want
to keep a separation between the code and the pipeline definitions you can select this method instead of having the
codefresh.yml file in the same place as the source code.
The url that you enter must be a public url of a raw YAML file. For example, if you want to add a link to a yaml file located in a public Github repository, you can use the ‘Raw’ option from the editor menu:
This way you can use a single
codefresh.yml file for different pipelines even in different repositories.
Switching between YAML and GUI steps
Once you switch to YAML mode, Codefresh will convert the existing GUI steps in the respective YAML syntax. This way you can easily upgrade a pipeline to YAML mode, after using the GUI steps and keep your custom commands. Note, that the opposite (going from YAML to GUI steps) is not supported or recommended. You will have start over if you switch to GUI mode.
You can easily recreate the predefined GUI steps with the following yml file.
This file contains 4 steps named, MyAppDockerImage, MyUnitTests, PushingToRegistry, MyIntegrationTests. Steps in Codefresh can have arbitrary names.
The first step is a build step that creates a Docker image, using a Dockerfile that is located at the root folder of your repository. The image is tagged with the name of the branch. More information for other ways of tagging can be found in the variables documentation.
The second step is a freestyle step that runs your unit tests in the context of the image that was just created.
The third step is a push step that pushes the image with the same tag. Since we haven’t defined a registry explicitly, the integrated Codefresh registry is used instead.
Remember that all Docker images of successful builds are always pushed to the Codefresh registry. The push step here is shown for demonstration purposes. It can be removed and the image will still be uploaded to the Codefresh registry. The push step is mostly useful for external Docker registries.
The last step is another freestyle step that runs integration tests, again inside the Docker image that was created in the first step.
For more information, see the complete YML syntax.