Programmatic Creation of Codefresh Pipelines – Part 2

Programmatic Creation of Codefresh Pipelines – Part 2

7 min read

At Codefresh, we know that any CI/CD solution must be attractive to both developers and operators (SREs). One of the major advantages of Codefresh is the graphical user interface that includes dashboards for Kubernetes and Helm deployments. These graphical dashboards are very useful to developers who are just getting started with deployments and pipelines.

We realize, however, that SREs love automation and the capability to script everything without opening the GUI. To this end, Codefresh has the option to create pipelines in a completely automated manner. We have already seen one such approach in the first part of this series where pipeline specifications exist in a separate location in the filesystem outside of the folders that contain the source code.

External pipeline specifications
External pipeline specifications

In this blog post, we will see an alternative approach where the pipeline specification is in the same folder as the code it manages.

Creating Codefresh pipelines automatically after project creation

Imagine that you are working for the operations/SRE team of a big company. Developers are constantly creating new projects and ideally, you would like their pipelines to be created in a completely automated manner. Even though you could use the Codefresh GUI to create any pipeline for a new project by hand, a smarter way would be to make Codefresh work for you and create pipelines on the spot for new projects.

To accomplish this we will create a “Pipeline creator” pipeline. This pipeline is creating pipelines according to their specification file. Therefore, the sequence of events is the following:

  1. A developer commits a new project folder
  2. An SRE (or the same developer) also commits a specification file for a pipeline
  3. In a completely automated manner, Codefresh reads the pipeline specification and creates a pipeline responsible for the new project
  4. Anytime a developer commits a change to the project from now on, the project is automatically built with Codefresh

In all of these steps, the Codefresh UI is NOT used. Even the “pipeline creator” pipeline is created outside of the GUI.

The building blocks for our automation process are the same ones we used in the first part of the series.

  1. The integrated mono-repo support in Codefresh
  2. The ability to define complete Codefresh pipelines (including triggers) in a pipeline specification file
  3. The ability to read pipeline specifications via the Codefresh CLI

We have seen the details of these basic blocks in the previous blog post, so we are not going to describe them in detail here.

A Codefresh pipeline for creating pipelines

You can find all the code in the sample project at https://github.com/kostis-codefresh/dynamic-pipelines

First, we will start with the most crucial part of the whole process. The pipeline creator. Here is the spec.

version: '1.0'
kind: pipeline
metadata:
  name: dynamic-pipelines/pipeline-creator
  description: "Pipeline creator for pipeline-spec.yml files"
  deprecate: {}
  project: dynamic-pipelines
spec:
  triggers:
    - type: git
      repo: kostis-codefresh/dynamic-pipelines
      events:
        - push
      branchRegex: /.*/gi
      modifiedFilesGlob: '**/pipeline-spec.yml'
      provider: github
      name: my-trigger
      context: github-1
  contexts: []
  variables: []
  steps:
    main_clone:
      title: 'Cloning main repository...'
      type: git-clone
      repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
      revision: '${{CF_REVISION}}'
      git: github-1    
    pipelineCreator:
      title: Setting up a new pipeline programatically
      image: codefresh/cli
      commands:
        - echo Creating a pipeline from $CF_BRANCH/pipeline-spec.yml
        - codefresh create pipeline -f $CF_BRANCH/pipeline-spec.yml
  stages: []



This specification file has two very important lines. The first one is the line:

modifiedFilesGlob: '**/pipeline-spec.yml'

This line comes from the monorepo support and essentially instructs this pipeline to run only “when a file named pipeline-spec.yml is changed in any folder”. The name we chose is arbitrary as Codefresh pipeline specifications can be named with any name that makes sense to you. This line guarantees that the pipeline creator will only run when a pipeline specification is modified. We don’t want to run the pipeline creator when a normal (i.e. code) commit happens in the repository.

The second important line is the last one:

codefresh create pipeline -f $CF_BRANCH/pipeline-spec.yml

Here we call the Codefresh CLI and create the pipeline that will actually compile the project. There are multiple ways to locate the file that contains the pipeline, but for illustration purposes, we make the convention that the branch name is also the name of the project.

If for example a new project is created that is named my-project, the SRE can do the following:

  1. Create a branch from master named “my-project”
  2. Add a directory called my-project
  3. Add a pipeline specification under my-project/pipeline-spec.yml
  4. Commit and push everything

Then the pipeline creator will read the spec and create the pipeline.

Setting up the pipeline creator

Now that we have the specification for the pipeline creator we are ready to activate it by calling the Codefresh CLI (i.e. using the same method of creating pipelines as the creator itself).

codefresh create pipeline -f pipeline-creator-spec.yml

The pipeline creator now exists in Codefresh and will be triggered for any builds that match the glob expression we have seen in the previous part.

You can visit the Codefresh UI to verify that everything is as expected.

Pipeline Creator
Pipeline Creator

Notice also the trigger that clearly shows that the pipeline creator will only run if a pipeline-spec file (in any directory) is among the modified files.

Adding a new project

Now that the pipeline creator is setup we are ready to add a project. To do this we will create a subfolder (e.g. my-go-app) and add the source code and the pipeline specification for that project. Here is an example:

version: '1.0'
kind: pipeline
metadata:
  name: dynamic-pipelines/my-go-app-pipeline
  description: "Docker creation for sample GO app"
  deprecate: {}
  project: dynamic-pipelines
spec:
  triggers:
    - type: git
      repo: kostis-codefresh/dynamic-pipelines
      events:
        - push
      branchRegex: /.*/gi
      modifiedFilesGlob: 'my-go-app/**'
      provider: github
      name: my-trigger
      context: github-1
  contexts: []
  variables:
    - key: PORT
      value: '8080'
  steps:
    main_clone:
      title: 'Cloning main repository...'
      type: git-clone
      repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
      revision: '${{CF_REVISION}}'
      git: github-1
    build_my_image:
      title: Building Docker Image
      type: build
      image_name: my-monorepo-go-app
      working_directory: ./my-go-app
      tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'
      dockerfile: Dockerfile
  stages: []

The pipeline itself is very simple. It only creates a Docker image (see the build_my_image step at the bottom of the file).

The important point here is the trigger. This pipeline will run only if the files changes are under the folder ‘my-go-app/**. This makes sure that this pipeline will only deal with that particular project and nothing else. Commits that run in other project folders will be ignored.

We now reach the magic moment. To create the pipeline for this project an SRE/developer only needs to:

  1. Create a branch named my-go-app
  2. Commit and push the code plus the specification

That’s it! No other special action is needed. The only interaction for developers/operators to get a pipeline running is via the GIT repo. The pipeline creator described in the previous section will detect this commit and automagically create a specific pipeline for the Golang project. The person that made the commit does not even need a Codefresh account!

Pipeline created automatically
Pipeline created automatically

Now any commit that happens inside the Go project, will trigger this pipeline (in our case it just creates a Docker image)

With this approach, anybody can keep adding projects along with their pipelines. The example monorepo has 3 projects. Here is the file layout:

Monorepo layout
Monorepo layout

And here is the view inside Codefresh:

All pipelines created automatically
All pipelines created automatically

Future improvements

In this second part of the series, we have seen how you can automatically create pipelines for multiple projects just by committing their pipeline specifications. In future posts, we will also explore the following approaches…

For illustration purposes, we used the convention where the name of the project is also the name of the branch that contains the pipeline. This makes the pipeline creator simple to understand but is not always practical. A better approach would have been to have a script that looks at all the folders of the project and locates the spec that was changed. This would also make it possible to create more than one pipeline with a single commit.

Also for illustration purposes, we assume that each project comes with its own pipeline spec. An alternative approach would be to have a simple codefresh.yml inside and then have the pipeline-creator convert it to a spec on the fly. This would make the creation of pipelines easier for developers that are familiar with Codefresh YAML but not the full spec. For bigger organizations, it would also be possible to have a templating mechanism for pipeline specifications that gathers a collection of global and “approved” pipelines and changes only the values specific to a project.

Finally, the present article shows only what happens when a project is initially created and the pipeline appears in the Codefresh GUI for the first time. A smarter approach would be to also recreate existing pipelines, so that the operator can change several settings after the initial pipeline creation.

New to Codefresh? Create Your Free Account Today!

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

5 thoughts on “Programmatic Creation of Codefresh Pipelines – Part 2

  1. I took a slightly different approach. I split out the triggers from the pipeline yaml since we have some pipelines with 30+ triggers and the pipeline yaml gets pretty unwieldy. In addition, I pulled out common attributes into base templates so you don’t need to specify them every time. Then a sync script pulls everything together to generate the full pipeline yaml and applies it.

  2. The last paragraph confirms what I’ve recently figured out and I’m honestly confused. Why does the pipeline API resource not accept a patch (I see there are other API resources that do)? Put another way: I can edit a pipeline in-place via the web UI but to change a trivial meta attribute like “description” via the API/CLI requires a destroy+create operation — why? It breaks the relationship between past logs and the pipeline.

    Is this functionality planned / in the backlog?
    Or was this intentional (and if so, why)?

    If I’m going to define this stuff in code then I’d like to be able to enforce it — ideally with an idempotent “apply” API call. I see there are other API resources that support a patch op. Why is the linchpin resource “pipeline” the black sheep? Not trying to hate here. This is def not a show-stopper for me. I’m a recent user and on the whole it’s been a pretty great experience but this issue just seems like a glaring blindspot.

    1. Hello. The last paragraph says that updating a pipeline is not described in this tutorial, not that it cannot be done.

      You can update a pipeline in place with the replace command. No need to destroy/create

      codefresh replace pipeline my-project/my-pipeline -f my-pipeline-spec.yaml

      As far as I know this is an idempotent operation. Would that work in your case?

      By the way, if you are an existing customer you can also open support tickets for feature requests. Feel free also use our community for more detailed Q/A https://community.codefresh.io/

      Let me know if I can help in any other way.

      1. ???? Awesome — testing shows “replace” behaves exactly as expected (and is idempotent). Not sure how I glossed over “replace”.

        ????‍♂️ #rtfm

        Thanks Kostis!

Leave a Reply

Your email address will not be published. Required fields are marked *

Comment

Build your GitOps skills and credibility today with a GitOps Certification.

Get GitOps Certified

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments