Create your FREE Codefresh account and start making pipelines fast. Create Account

CI/CD Pipelines for Microservices

8 min read

Moving from monoliths to microservices and from Virtual Machines to Kubernetes clusters are two closely related migration paths. Using Kubernetes clusters enables the easy deployment of microservices, and adopting microservices becomes a lot easier when containers are used for the packaged delivery.

If you look at our features page, it is very easy to understand why Codefresh is the first (and only) CI/CD platform for containers and Kubernetes. Currently, Codefresh is the only solution that includes:

All these features show clearly that Codefresh is the perfect solution for CI/CD if you have already adopted containers and Kubernetes. However, they don’t tell the story of the monolith-to-microservice adoption.

In this article, we will see how microservices change the way CI/CD pipelines are organized and why Codefresh is specifically designed with microservices in mind.

Organizing pipelines for monolithic applications

In the past, pipelines for monolithic applications tended to share the same characteristics of the application they were building. Each project had a single pipeline which was fairly complex and different projects had completely different pipelines. Each pipeline was almost always connected to a single GIT repository.

Pipelines for monolithic applications
Pipelines for monolithic applications

The complexity of each pipeline was detrimental to easy maintenance. Pipelines were typically controlled by a small team of gurus who are familiar with both the internals of the application as well as the deployment environment.

For each software project, operators are taking care of the pipeline structure while the developers are only working with the source code (going against the DevOps paradigm where all teams should share responsibility for common infrastructure and collaborate on shared problems).

Pipeline size and complexity is often a huge pain point. Even though several tools exist for the continuous integration part of a monolithic application, continuous deployment is a different matter completely which forced a lot of companies to create their own custom in-house scripts for taking care of deployment.

As the number of applications evolved within an organization, the management of pipelines became a bigger issue as well. Usually, most pipelines end up as a spaghetti mix of python scripts, configuration management tools (e.g. Puppet and Chef) sprinkled with some bash scripting all over the place.

This messy situation is very common within organizations that have not adopted microservices yet and the only reason that it endures is that most developers are still working on a single application so they never feel the pain of pipeline maintenance. That job falls on the operators who are typically spending most of their time fixing pipelines (and creating new ones for new projects) assuming that this is the status quo.

Scalability issues with microservice pipelines

Microservices have of course several advantages regarding deployment and development, but they also come with their own challenges. Management of microservice repositories and pipelines becomes much harder as the number of applications grows.

While a company might have to deal with 1-5 pipelines in the case of monolith applications (assuming 1-5 projects), the number quickly jumps to 25 if each monolith is divided into 5 microservices.

These numbers are different per organization. It is perfectly normal for an application to have 10 microservices. So at a big organization that has 50 applications, the operator team is suddenly tasked with the management of 500+ pipelines.

Moving to microservices
Moving to microservices

This sudden explosion in numbers prohibits working manually with pipelines anymore. Several CI solutions are not even prepared to work with such a high number of pipelines.

Here is where we reach the biggest pitfall regarding pipeline management in the era of microservices. Several companies tried to solve the problem of microservice pipelines using shared pipeline segments.

Shared pipelines are complex
Shared pipelines are complex

In theory, this sounds like a good idea:

  1. Operators are locating the common parts of pipelines with applications
  2. A shared pipeline segment registry is created that holds all those common parts
  3. Pipelines in existing projects are re-engineered to depend on the common segments
  4. New projects must first examine the library of common pipeline segments and choose what is already there

The final result is that a single pipeline is actually composed of two types of steps, those common to other pipelines and those that are specific to that project only.

This has lead to the development of several solutions which attempt to centralize common pipeline parts and re-use them in the form of “libraries” within software projects. The issue here is that this approach requires a very large time investment as well as a disciplined team that can communicate and cooperate on the following factors:

  1. Detecting which pipeline segments are indeed common,
  2. Keeping the library of common pipeline segments up-to-date,
  3. Disallowing copy-pasting of pipelines,
  4. Development of brand new pipelines when needed,
  5. Initial setup and pipeline bootstrap for each new project created.

Unfortunately, in practice, as the number of microservice applications grows, teams find it very hard to keep all these principles in mind when creating new projects.

Reusing pipelines for Microservice applications

Codefresh is the first CI/CD solution for microservices and containers. Because we are not burdened with any legacy decisions, we are free to define a new model for Codefresh pipelines which is focused specifically on microservices.

The basic idea is that all microservices of a single application have almost always the same life-cycle. They are compiled, packaged and deployed in a similar manner. Once this realization is in place we can see that instead of having multiple pipelines for each microservice (where each one is tied to a GIT repository), we have instead a single pipeline shared by all microservices.

Monolith split to microservices
Monolith split to microservices

The impact of this design cannot be understated. First of all, it should be clear that there is no need for sharing pipeline segments anymore. The whole pipeline is essentially the re-usable unit.

This makes pipeline construction very simple.

The biggest advantage, however, is the way new projects are created. When a new microservice is added in an application, the pipeline is already there and only a new trigger is added for that microservice. Notice that the pipeline is not connected to any specific git repository anymore. All information for a repository is coming from the git trigger that started this pipeline.

This is exactly what our recent feature launch was based on. As an operator you can bootstrap a new project by quickly adding a new trigger on an existing pipeline:

Sharing a single CI/CD pipeline
Sharing a single CI/CD pipeline

This is the fastest way possible to bootstrap a new project. As the number of microservices is growing, the only thing that is growing is the list of triggers. All pipelines are exactly the same.

Creating a new pipeline is needed only if the whole deployment process is different for a new project. This is an extreme scenario however, as with containers this will happen very rarely. Once you package an application in a Docker image, the underlying implementation is not really important anymore. Docker images from different programming languages can be handled in the same manner.

Case study: Codefresh on Codefresh

Now that we have explained the theory behind microservice pipelines, we can look at a solid example. And one of the natural examples is Codefresh itself. A very popular question we get from our customers is how our own workflow is structured. It shouldn’t be a big surprise that we also use Codefresh internally to manage Codefresh.

Codefresh running on Codefresh
Codefresh running on Codefresh

The Codefresh platform is itself a set of microservices that run on Kubernetes. We have about 25 microservices, but they are easily managed by a handful of pipelines.

In fact, most of the heavy work is done by two pipelines:

  • A pipeline called “CI” which packages each microservice.
  • A pipeline called “CD” which deploys each microservice.

That’s it! Even if tomorrow our platform has 50, 500, or 5000 microservices the number of pipelines will still be 2. Every new microservice will be handled by simply adding a new trigger to those pipelines.

By the way, our pipelines are public, so feel free to adopt them for your own infrastructure (or at least study them and get some ideas on possible workflows).

Let’s see them in order…

A CI pipeline for microservices

First, we have the Continuous Integration pipeline. This pipeline is connected with git triggers to each of our microservices.

CI pipeline for microservices
CI pipeline for microservices

The pipeline steps do the following:

  • Check out the code for each service
  • Check the software version and whether this is a Pull request or not
  • Download extra dependencies for testing
  • Run linting (i.e. syntax checks) on the source code
  • Run unit tests
  • Run integration tests using Docker compositions
  • Generate test reports.
  • Build deployment docker images
  • Run security scans using Twistlock
  • Push images to the internal docker registry
  • Create Helm packages

The pipeline definition has several conditionals so not all steps are running at all builds. Some of them are skipped depending on the nature of the commit.

The pipeline is connected via git triggers to all microservices, so whenever a commit happens it runs automatically and prepares both a Docker image and a Helm chart for each release candidate.

The important point is that this pipeline does NOT refer to any specific git repository, nor it is connected to one. It is a generic CI pipeline that is reused by all microservices depending on the trigger that calls it.

This makes pipeline maintenance very easy, as there is only a single definition to maintain. It also means that all best practices within a company (e.g. security scanning or quality control) are automatically applied to all projects and teams without any extra effort.

A CD pipeline for microservices

The next step is to run the Continuous Deployment pipeline for each microservice:

CD pipeline for microservices
CD pipeline for microservices

This pipeline is launched automatically as part of a “promotion” which takes place when the Codefresh team is making an official release for a microservice. It does the following:

  • Checks that no other deployments are happening in parallel for the same service
  • Waits for manual approval
  • Installs the previously prepared Helm package (created in CI pipeline)
  • Commits back to the git repo the dependencies list of the app
  • Creates a mark in New Relic for monitoring purposes
  • Sends a message to our Slack channel to show the status of the deployment

Again, if you read the pipeline definition you will see that it is not tied to any specific git repository. Instead, it is reused for all microservices that need it.

The advantages of using CI/CD pipelines for microservices in this pattern should now be clear to you. Instead of spending time with a mess of pipelines and a big number of “reusable” pipeline segments or shared libraries, with Codefresh you can centralize all the CI/CD logic in just two pipelines.

The icing on the cake is that if tomorrow we need to add a new microservice in the mix we only need to add a new GIT trigger for it in the existing pipelines. It doesn’t get any easier than this!

Adding a new CI/CD pipeline for a microservice
Adding a new CI/CD pipeline for a microservice

Ready to try Codefresh and start creating your own CI/CD pipelines for microservices? Create Your Free Account Today

Kostis Kapelonis

Kostis is a software engineer/technical-writer dual class character. He lives and breathes automation, good testing practices and stress-free deployments with GitOps.

10 responses to “CI/CD Pipelines for Microservices

  1. Michael . Kolb says:

    How do you export environment variables into the pipeline when you use a single pipeline with multiple triggers? I have environment variables that are used to name and tag the docker images that i build, and unique S3 file paths for storage of Terraform State files. I use a lot of variable interpolation via a cf_export step so that i have significant code re-use of pipeline yaml code. I also inject SSH and GPG keys at container runtime so that secrets are never stored in the images, and I wouldn’t always want to use the same SSH or GPG keys per microservice. How would you handle exporting environment variables and importing SSH/GPG keys into a common pipeline definition?

    1. Kostis Kapelonis says:

      Very good question!

      I would use shared configuration ( where each microservice has a different set of variables/parameters and then fetch the values programmatically with the Codefresh CLI inside a pipeline (see last example in that page)

      But specifically for secrets, for complex cases I think they should not be part of CI/CD, but instead secrets should be fetched during runtime using Hashicorp vault, Lyft Confidant and other similar solutions.

  2. Michael Kolb says:

    We follow a GitOps model were everything related to our project code is in our Git repository. Since a shared-configuration is outside of the Git repo this process would not work in our environment. Can the cf_export command be invoked in a bash shell script of a freestyle step? I am looking for a method to inject these unique repository environment values into the pipeline at runtime, since i think the microservices model is amazing. And, I want to implement it in our environment.

  3. Mark Johnson says:

    I’m not sure if the same “simplicity” applies with a monorepo? We use multiple pipelines for this which works well for the task. Plus several microservices break-away from the norm (not creating lambda functions), such as updating MongoDB Stitch functions.

    1. Kostis Kapelonis says:

      Well the only thing that changes with the monorepo is the trigger. Of course maybe you need some extra effort to make sure that your pipeline is indeed reusable for all microservices placed in the monorepo. But other than that I don’t see any other issue. Did you see already the post about monorepo support? 🙂

      1. Mark Johnson says:

        We already use monorepos, but with a pipeline for each service with its own codefresh..yml definition; most are almost identical but some are quite different. It would be good to see if we can potential reduce the number of yml files.

  4. Has anybody got this to work. It is exactly what we need, but I don’t get how we manage root keys for each pipeline. Yes, we can an do use Hashicorp Vault, but we would still need a token to access vault and we would want that token restricted on what it can access, so at a minimum a token per application. I don’t believe we can use shared configuration for that, because there is no RBAC on access to shared secrets. I think I’m missing a step somewhere, but definitely this is the approach we need to adopt.

    BTW no longer works. Is there a new link to the example CF pipelines

    1. Kostis Kapelonis says:

      Hello. We updated the link and it should not be broken anymore. Sorry for the confusion

Leave a Reply

* All fields are required. Your email address will not be published.