I have always liked to help my customers. In my previous job at Electric Cloud, my first three customers asked me the same question on how to clean old jobs. I realized we had a gap in our product, so I wrote a plugin to help them manage their CI server. The EC-Admin plugin (https://github.com/lrochette/EC-Admin) has grown over the years as customers were presenting new requests. It became quite famous among my customers, my field teammates, and particularly the support team. This might be part of the reason Vidhya, Head of Customers at Codefresh and my former support colleague, asked me to join her at Codefresh.
So when in my first week she asked whether I could help a customer and write something for them, I jumped at the opportunity. Hey, what’s more fun than solving a customer issue, exploring my new platform, and writing a plugin?
If you’re reading this blog, I assume you’re somewhat familiar with Codefresh. Codefresh is a modern DevOps automation platform with comprehensive CI/CD pipelines and custom deployment Dashboards. The platform is designed to build, test and deploy cloud-native applications. Each step is built on top of a container image. Have a look at our documentation https://codefresh.io/docs/docs/getting-started/create-a-basic-pipeline/ for more information.
The Problem
The customer was looking for a way to manage pipelines more easily in an automatic way instead of their manual, UI-heavy process. A pipeline is composed of two main parts:
- A core that contains the definition of its stages and steps that drive its execution.
- A set of triggers. Triggers are bits of code to manage when a pipeline should be started.
Saving those separately in files in their SCM allows the customer to provide a library of pipeline templates that can simply be combined with the required pre-defined triggers.
Step 1: Freestyle
The solution involved writing YAML, file management, and Codefresh CLI. To explore the problem and find an adequate solution, I decided to start with a simple “freestyle” step to test ways that I could make it work. A freestyle step is a step that runs commands on a container-based image of your choice.
I decided to use the “codefresh/cli” image as it contains the Codefresh CLI (Command Line Interface) and the tool yq. yq stands for “YAML query” and is a lightweight and portable command-line YAML processor. Have a look at https://mikefarah.gitbook.io/yq/ for more information.
Tip: there is not a lot of documentation on yq, but it’s built on top of jq (the JSON version). Most jq options and mechanisms work for yq, and jq is a lot more documented. Check https://stedolan.github.io/jq/ for details.
So my first step looked like
steps:
...
merge: # name of the step
title: “merge pipeline and triggers” # long description
image: codefresh/cli # image to use to run the step
commands: # list of commands to run
- |-
Loop on trigger file
yq -m pipeline.yaml trigFile.yml # merge pipeline and trigger file
End loop
# upload full pipeline on Codefresh
codefresh cli create pipeline -f pipeline.yml
As you can see, nothing super complex. It is a simple loop on the list of triggers to merge them one by one with the pipeline. The hardest part was trying to find the right syntax and correct syntax for yq to merge the YAML pipeline and each trigger correctly.
Once the freestyle step was working, I could have passed that bit of code to the customer. However, this would have potentially resulted in more long-term work for the entire support team and our customers because each bug would have to be addressed separately.
Step 2: A private typed step
To solve that copy/paste issue, I pursued my effort and decided to transform my freestyle step into a typed step. A typed step is basically a plugin that allows you to interact with any third-party system.
This allowed me to share the step more easily through our marketplace, a public repository of plugins created by Codefresh in collaboration with our partners and customers.
This decision also made it possible to keep the code available for anybody to improve or fix a bug. You can check the latest version here.
The following sections will describe the use of the step within your Codefresh pipeline.
2.1 Define your Docker image
I re-used the codefresh/cli and simply added a bash file that contains a more robust version of the code I created for my first iteration of the freestyle step.
2.2 Step definition
When you want to create a new typed step, the easiest way is simply to pick up the step.yaml from another typed step and adapt it to your needs. You can find a guide within our documentation. Existing steps can be found in our official GitHub repository: https://github.com/codefresh-io/steps/
Some of the information is pretty obvious and includes name, version, and maintainers.
You also define your inputs and can even mark some as compulsory so that the Codefresh pipeline will make sure the user passes the information required by the step.
I also started marking it non-public (only visible in my account) and not official. This allows you the freedom to develop at your own pace without worrying that someone else will pick it up before it’s ready and potentially damage their environment.
To test easily, my step name was originally limited to my account “laurent-cf/pipeline-trigger-merge” and the image was in my private Docker registry.
After building and pushing the image, creating the step, it was time to test. My new step was therefore a little simpler from an end-user perspective. Instead of having a block of code, it now has just a simple call with parameters. I would compare it to using a function in a programming language.
merge:
type: laurent-cf/pipeline_trigger_merge
title: "Creating pipeline"
working_directory: ${{clone}}
stage: clone
arguments:
TRIGGERS: foobar trig1.yml trig2.yml TRIGGERS
SPEC: pipeline.yml
It’s even simpler as the step was available in my account step list. The users in your organization can now use this step very easily. They can see an example and documentation to include it in the “My Steps” section of the workflow definition:
Publish to the Official marketplace
For most of you, you would stop at the previous step. But for me as an employee at Codefresh, I had the opportunity to make it public and official. It was a super simple process:
- Change the image name to the official codefreshplugins/pipeline-trigger-merge
- Push to the public Codefresh registry
- Mark as official and public, using the matching fields in the YAML description.
- Create a PR to get feedback and merge it into the master branch
- The official plugin process takes over
Now my first step is available for all of you to use in the marketplace.
Or at https://codefresh.io/steps/step/pipeline-trigger-merge
Next steps
The next thing I’d like to do is to convert the Bash code into a Go program. It’s really a personal preference as you can write your code in any language as long as the underlying image can handle it.
I started a new branch for that effort, but I have not yet found the correct way to merge my YAML files so far.
I will also explore Step Templating to manage the set of trigger files, instead of using a space-separated list. Technically, it’s in my step definition but I’m not making use of it.
This was my first step, so I’m sure there are a lot of things that could be done better. Feel free to reach out at [email protected] or create a PR to let me know.
If I were able to create something in my first few weeks at Codefresh, it means others can do it too! I was baffled by how simple the whole process was. It’s not my first rodeo (Texas reference ;)) but it’s certainly the easiest mechanism I’ve used to create a full-fledged plugin in my career. The most difficult decisions were to decide what image and language to use. My colleagues are too young to see the beauty of Perl 😉
So give it a try and register if you’re not already a Codefresh user. Feel free to reach out to our support team if you have any questions.
Hopefully my next plugin, an AWS CDK wrapper, will be a lot better.