Codefresh vs. GitlabCI

If you have selected Gitlab as your git provider, you may automatically think about selecting GitlabCI as your CI/CD solution as well. Using the same vendor for source control and CI/CD might seem natural, but is not always the best combination. In fact, choosing a “good-enough” option just because it is part of the same solution is unwise in the long run if it doesn’t cover your needs.

Gitlab is one of the supported GIT providers in Codefresh, so another valid combination is using Gitlab for source control and Codefresh for CI/CD. In this article, we will look at the advantages of Codefresh compared to the GitlabCI platform.

Update: See Gitlab’s response to this comparison here

Content was updated as both products changed since initial writing

Codefresh is GIT agnostic, GitlabCI has native support only for its own repos
Codefresh allows you to create multiple pipelines per repo, while GitlabCI is constrained to just one
Docker builds are much faster with Codefresh
GitlabCI needs special configuration to share information between pipeline phases
The Docker registry in Codefresh is fully automated
The Docker registry in Codefresh is much more detailed
Codefresh has explicit support for Kubernetes deployments
Codefresh has explicit support for Helm deployments

Codefresh is GIT agnostic, GitlabCI is not

This is probably the biggest difference between Codefresh and GitlabCI. Codefresh has native support for all popular Git providers such as Github, Gitlab, Bitbucket, Azure Devops Git, Atlassian Stash and can work with both cloud and hosted variants of them. All Git providers enjoy equal support in Codefresh and all features (git triggers, monorepo changes, checkouts, webhooks etc) work in a similar manner.

GitlabCI has native support only for Gitlab repositories. If your source code is located somewhere else you need to import it into Gitlab, basically creating a second copy. Specifically for Github, there is also partial support for “mirroring” where a second clone of your git repository is placed into Gitlab and all upstream changes are automatically synced. But the communication is one way. If you make changes into Gitlab they won’t be synced back to Github. The process is very cumbersome as essentially you have two repos to manage and they can easily suffer from drift issues.

At the time of writing there is a whole epic open (with 10+ issues), so at least Gitlab is aware of this limitation. But until this epic is implemented Codefresh is the only viable solution if you work in an organization that is using multiple GIT providers.

We find that several companies want to use other Git providers apart from Gitlab, so Codefresh has a major advantage on the topic of GIT support.

Codefresh allows you to create multiple pipelines per repo, while GitlabCI is constrained to just one

This was another big revelation during my time with GitlabCI. In GitlabCI you can only define a single pipeline per project. That’s it! There is no way to add more pipelines. In the screenshot below, you can see there is really only one “Run Pipeline” button:

A nontrivial application will need more than one pipeline. A common set of pipelines would be:

A pipeline that runs on every commit that packages code and probably creates a Docker image.
A pipeline the deploys to an environment (e.g. prod or staging).
A pipeline that runs load testing against a specific environment.
A pipeline for pull requests that might run code quality or other security tools.
A weekly build for cleanups or maintenance actions.

GitlabCI offers the capability to have conditionals in a pipeline, so in theory, you could create a really huge pipeline that acts as a superset of all the things that you might ever need. But this is not ideal, and there is even an issue that has been open since 2016 for this particular problem.

Codefresh does not suffer from such limitations. Even though Codefresh also allows you to define pipeline conditionals, you are still free to define multiple pipelines for every workflow imaginable.

In a similar manner, the git trigger capabilities in Codefresh can cover all webhook events sent by the git provider resulting in pipelines that can run against specific git events. This can be done right from the GUI with no special configuration in the build yaml file.

This limitation of GitlabCI essentially forces you into having a huge gitab-ci.yml file with multiple conditional steps that select the “proper” workflow when an event is triggering the pipeline. For a really really complex workflow, this can quickly get out of hand.

Codefresh also supports parent/child relationships which GitlabCI currently does not.

Docker builds are much faster with Codefresh

Codefresh has several types of built-in caching and two of them are actually specific to Docker builds.

One of the most powerful Codefresh features is the zero-configuration distributed docker layer cache. All Codefresh build nodes share it which means that all subsequent builds will use the previously cached layers. It works exactly as you would expect from the Docker layer cache in your local workstation:

This cache is completely automatic. There is nothing to setup or configure. It works in all versions of Codefresh (even the SAAS one). GitlabCI has no such cache and no matter how many times you build a docker images, it will rebuild everything from scratch. This means that Docker builds are very slow in GitlabCI and depending on the size of your images (and number of layers) the amount of time you lose by waiting for builds can become a bottleneck.

Here is an actual example where the same project is packaged with Codefresh and GitlabCI without actually changing anything.

Codefresh is very fast because the second type of cache has kicked-in and has detected that this Docker image is already built.

Codefresh detects an existing Docker image

At the time of writing there is an open issue for Docker cache in GitlabCI. But as things are now, just by building a project with Codefresh you will get much faster builds.

GitlabCI needs special configuration to share information between pipeline phases

The pipeline phases in GitlabCI are called “jobs” (in Codefresh they are called “steps”). These run in a completely independent manner so everything that needs to be shared between pipeline phases must be defined explicitly.

To accomplish this, GitlabCI introduces a special syntax for two different concepts: artifacts and caching.

If you need to transfer something from one phase to another you need to explicitly define it as cache. For example, let’s say you have two GitlabCI jobs that both use node modules. Unless you define the folder as cache, both of them will re-download everything.

The correct syntax in GitlabCI would be:

 
cache:
 paths:
    - node_modules/
setup:
 stage: setup
 image: node:7.9
 script:
    - npm install

test:
 stage: test
 image: node:7.9
 script:
    - npm test

If you wish to just move things from one job to another you are expected to use artifacts. For example, if you create a jar file in one job and expect to find it in the next one, you first need to add the following to your GitlabCI file:

 
build:
 stage: build
 image: maven:3.3.9-jdk-8
 script:
    - mvn package
 artifacts:
    paths:
     - target/*.jar

The distinction between caching and artifacts is not always very clear, forcing even the official documentation to offer advice on best practices for what to consider cache and what to consider an artifact.

So how does Codefresh deal with this issue?

In Codefresh, in order to transfer information from one step to the other, you do (drumroll please)….absolutely nothing. In Codefresh all pipeline phases share a common volume (which includes the code of your project). So everything that is created in the project folder is automatically available to the next phase.

This means that you don’t need to do anything special to cache node modules. In Codefresh yml the same thing would be:

 
setup:
 title: setup
 image: node:7.9
 commands:
    - npm install

test:
 title: test
 image: node:7.9
 commands:
    - npm test

The same goes for “artifacts”. There is no need to define something explicitly for binaries, test reports, or anything else that you wish to be visible to all pipeline steps.

 
build:
 title: build
 image: maven:3.3.9-jdk-8
 commands:
    - mvn package

The end result is that Codefresh pipelines are much more compact and readable. You can find more information about Codefresh pipelines in the documentation.

The Docker registry in Codefresh is fully automated

Both Codefresh and GitlabCI offer a built-in Docker registry that can be used for storage of Docker images. The Docker registry offered by GitlabCI acts as any other Docker registry and in order to use it, you have to explicitly call Docker commands in your yml file.

For example, in order to push an image in GitlabCI you need to do the following:

 
create-image:
 stage: create-image
 image: docker:stable
 services:
    - docker:dind
 script:
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"

This yml syntax essentially reflects what you would write in locally on your workstation. Codefresh offers much better support for its own internal registry.

In order to push your image to the Codefresh registry you do (drumroll please)….absolutely nothing. By default, all your Docker images that are produced by your builds are automatically pushed to the internal registry. You don’t add anything in your yml file to enable this functionality.

And before you ask about wasted space, know the Codefresh pricing is independent of the number of Docker images you have, so don’t worry about overusing the internal registry. As we will also see in the next section, it is very easy to locate images even when their number is very big.

If you want to use an external registry, Codefresh helps you even further by allowing you to declare all your registries at your account level (including authentication details).

Once you want to use an external registry you only need a push step:

 
build_image:
   title: Building DockerImage
   type: build
   image_name: my-app:master
push_name:
 type: push
 title: Pushing to ECR
 candidate: ${{build_image}}
 tag: latest
 image_name: codefresh/my-app
 registry: my-ecr-registry

Notice the complete lack of login instructions in Codefresh. You only define the registry name as it was defined in the GUI. With GitlabCI you would need to replicate the Docker login commands as before.

In summary, working with the integrated Docker registry is completely automatic with Codefresh, while GitlabCI needs the exact push instructions.

The Docker registry in Codefresh is much more detailed

We have seen in the previous section how easy it is to use the Codefresh internal registry and talked about the yml syntax for each platform. Once you have many images in your Docker registry you need a way to find them and catalog them.

The GUI view of the registry in GitlabCI is very sparse:

You only see the tag, the size, and the date of the image. There is no other option available (which might make sense as we have already seen that the registry in GitlabCI is tied to a single project).

Codefresh, on the other hand, gives you several more options. As a starting point, you can narrow down your view using multiple filters

You can also do an analysis of the layers of each image to gain insights of the size:

And most importantly of all, you can annotate your Docker images with your own metadata. For example, you can mark images that have passed security scans, or code quality gates, or load testing approval. The metadata can be added either directly from the GUI or via the pipeline syntax.

This makes the Codefresh Registry very powerful, making it a central source of truth for developers, QA, operations and any other stakeholder of a project.

In summary, the integrated GitlabCI registry lacks a lot of features offered by the Codefresh version.

Codefresh has explicit support for Kubernetes deployments

Easy Kubernetes integration is one of the major features behind the development of Codefresh. Kubernetes support is one of the highlights of Codefresh functionality.

When you define a Kubernetes cluster in Codefresh, you get access to two major features: First, you have a built-in Kubernetes dashboard that shows you namespaces/pods and deployments:

Second, Codefresh allows you to deploy a Kubernetes service with just the GUI, which is a great way to quickly deploy demo apps.

In vanilla GitlabCI there is no such thing. Apart from the deploy boards, there is no other GUI that can help you with your first steps in Kubernetes management.

Codefresh has explicit support for Helm deployments

Helm is a package manager for Kubernetes that allows you to group multiple application into charts.

GitlabCI has some basic support for Helm. It allows you to easily install Helm on your Kubernetes cluster (i.e. the server component called Tiller).

Once Tiller has installed it is also very easy to use Helm to install packages.

Codefresh, on the other hand, takes Helm integration to the next level by providing:

an integrated Helm repository for all accounts,
a graphical app browser that can access both the internal and external helm repos,
and a Helm dashboard for monitoring deployments and even rolling them back.

None of these is offered by GitlabCI and having an internal Helm repository is really crucial for using Helm in the first place (as the built-in Docker registry is useful for Docker images).

Codefresh gives you a built-in Helm repository that you can use for your own charts. You can also connect other chart repositories and create an application catalog.

Once you install a Helm chart in your Kubernetes cluster you have full visibility on version history, active services, chart values, etc.

However, the most impressive capability is that you can rollback a version right from the GUI. And in this case, rollback means the Helm rollback (i.e. going to a previous version of the chart).

In summary, Helm support in Codefresh is much more extensive than GitlabCI and the presence of a built-in Helm repo and dashboard are very important factors for using Helm as part of the CI/CD process.

Autodevops is a hit-and-miss feature

In the first version of this comparison I didn’t say anything about the Auto-devops feature of GitlabCI. The reason was that I consider this feature experimental and it would not make sense to comment on its stability.

However, seeing how GitlabCI uses it as a direct discussion point on its own comparison page I couldn’t resist on reviewing it as well. I tried autodevops on 3 sample projects (Java, Python, Ruby on Rails) and it worked only on the Java project.

In the Python project it failed to find any tests (even thought the repository has unit tests).

Autodevops fails with a Python repo example

It turns out that Auto-devops is based on the same ideas as Heroku buildpacks, which are good for starter projects, but cannot work in all possible cases.
In my Ruby project I got a completely different error:

Autodevops fails with a Ruby project example

Now of course I could track down these errors and try to fix them, but that would beat the whole purpose of “autodevops”. People that know enough to fix those kind of errors can also create their own pipelines manually.

It is also interesting to note that in the past Codefresh had its own “autodevops” feature. We tried as well to detect the kind of source code contained in a repository and tried to suggest a pipeline. But it didn’t work as reliably as we wanted and we scrapped this feature in order to avoid giving false information to our customers.

So while Auto-devops is a very nice idea, in practice it is so unpredictable that I wouldn’t consider it an advantage of GitlabCI over any other CI platform.

Conclusion

In summary, Codefresh is a solution designed specifically for microservices and containers, while GitlabCI is a generic platform where Docker support is an afterthought and Helm support is in its infancy. If you have already switched to Docker/Kubernetes and especially Helm, Codefresh brings much more value to the table.

At the time of writing the major advantages of GitlabCI over Codefresh are the built-in NPM and Maven registries and the capacity to define artifacts in a pipeline.