GitOps deployments

Learn how to deploy with Codefresh and ArgoCD

Apart from traditional push-based Helm deployments, Codefresh can also be used for GitOps deployments.

What is GitOps

GitOps is the practice of performing Operations via Git only. The main principles of GitOps are the following:

  • The state of the system/application is always stored in Git.
  • Git is always the source of truth for what happens in the system.
  • If you want to change the state of the system you need to perform a Git operation such as creating a commit or opening a pull request. Deployments, tests, and rollbacks controlled through git flow.
  • Once the Git state is changed, then the cluster (or whatever your deployment target is) state should match what is described in the Git repository.
  • No hand rolled deployments, no ad-hoc cluster changes, no live configuration changes are allowed. If a change needs to happen, it must be committed to Git first.

GitOps deployments have several advantages compared to traditional imperative deployments. The main one is that the Git repo represents the state of the system, and Git history is essentially the same thing as deployment history. Rollbacks are very easy to perform by simply using a previous Git hash.

Even though GitOps is not specific to Kubernetes, current GitOps tools work great with Kubernetes in the form of cluster controllers. The GitOps controller monitors the state of the Git repository and when a commit happens, the cluster is instructed to match the same state.

Codefresh has native support for GitOps including a graphical dashboard for handling your GitOps deployments:

The GitOps dashboard

The GitOps dashboard

This guide will explain how you can use GitOps for your own applications.

Setting up your Git repositories

One of the central ideas around GitOps is the usage of Git for ALL project resources. Even though developers are familiar with using Git for the source code of the application, adopting GitOps means that you need to store in Git every other resource of the application (and not just the source code).

In the case of Kubernetes, this means that all Kubernetes manifests should be stored in a Git repository as well. In the most simple scenario you have the main repository of your application (this is mostly interesting to developers) and a second Git repository with Kubernetes manifests (this is more relevant to operators/SREs).

As a running example you can use:

The application code repository contains the source code plus a dockerfile. You can use any Git workflow for this repository. We will set a pipeline in Codefresh that creates a container image on each commit.

The configuration repository holds the kubernetes manifests. This is one of the critical points of GitOps

  • The configuration repository holds the manifests that are also present in the Kubernetes cluster
  • Every time a commit happens to the configuration repository the cluster will be notified to deploy the new version of the files (we will setup a pipeline for this)
  • Every subsequent configuration change should become a Git commit. Ad-hoc changes to the cluster (i.e. with kubectl commands) are NOT allowed

We also have a third Git repository for pipelines, because pipelines are also part of the application.

Before continuing fork all 3 repositories in your own GitHub account if don’t have already your own example application.

Connecting ArgoCD and Codefresh

GitOps deployments are powered by ArgoCD so you need an active ArgoCD installation in your cluster to take advantage of the GitOps dashboard in Codefresh.

Follow the instructions for connecting ArgoCD to Codefresh and creating an ArgoCD application

Creating a new ArgoCD application in a Codefresh environment

Creating a new ArgoCD application in a Codefresh environment

The options are:

  • Name - User defined name of the Codefresh environment dashboard
  • Project - A way to group/secure applications. Choose default if you have only one project in ArgoCD.
  • Application - name of application
  • Manual/automatic sync - If automatic when a git commit happens, a deployment will automatically take place.
  • Use schema - Kubernetes manifests will be checked for correctness before deployed to the cluster
  • source repository - Git repository that holds your Kubernetes manifests
  • revision - Revision to be checked out when a deployment happens
  • path - folder inside the Git repository that should be searched for manifests (if your Git repo has multiple applications)
  • cluster - Kubernetes cluster when deployment will take place
  • namespace - Kubernetes namespace where the application will be deployed to
  • directory recurse - wether to check all folders in the Git repository for manifests in a recursive way.

For a sample application you can use the https://github.com/codefresh-contrib/gitops-kubernetes-configuration repository. Fork the project in your own GitHub account and use that link in the Source repository section.

Once you connect your application you will see it under in the GitOps application screen in the Codefresh UI.

Creating a basic CI pipeline for GitOps

Creating a CI pipeline for GitOps is no different than a standard pipeline that packages your Docker images, runs tests, performs security scans etc.

Basic CI pipeline

Basic CI pipeline

To take advantage of the GitOps dashboard facilities you also need to setup the correlation between the Docker image and the Pull Requests/issues associated with it. This correlation happens via annotations. The easiest way to annotate your image is by using the pipeline plugins offered by Codefresh for this purpose. Currently we offer the following plugins:

Here is an example Pipeline definition:

codefresh.yml

version: "1.0"
stages:
  - "clone"
  - "build"
  - "metadata"

steps:
  clone:
    title: "Cloning repository"
    type: "git-clone"
    repo: "my-github-username/gitops-app-source-code"
    revision: '${{CF_REVISION}}'
    stage: "clone"
  build:
    title: "Building Docker image"
    type: "build"
    image_name: "kostiscodefresh/simple-web-app"
    working_directory: "${{clone}}"
    tags:
    - "latest"
    - '${{CF_SHORT_REVISION}}'
    dockerfile: "Dockerfile"
    stage: "build"
    registry: dockerhub  
  enrich-image:
    title: Add PR info
    type: image-enricher
    stage: "metadata"
    arguments:
      IMAGE:  docker.io/kostiscodefresh/simple-web-app:latest
      BRANCH: '${{CF_BRANCH}}'
      REPO: 'kostis-codefresh/simple-web-app'
      GIT_PROVIDER_NAME: github-1
  jira-issue-extractor:
    title: Enrich image with jira issues
    type: jira-issue-extractor
    stage: "metadata"
    fail_fast: false
    arguments:
      IMAGE: docker.io/kostiscodefresh/simple-web-app:latest
      JIRA_PROJECT_PREFIX: 'SAAS'
      MESSAGE: SAAS-8431
      JIRA_HOST: codefresh-io.atlassian.net
      JIRA_EMAIL: [email protected]
      JIRA_API_TOKEN: '${{JIRA_TOKEN}}'

This pipeline:

  1. Checks out the source code of an application with the git-clone step
  2. Builds a docker image
  3. Annotates the Docker image with the Pull Request information provided by Github
  4. Annotates the Docker image with a specific Jira issue ticket

You can see the associated metadata in your Docker image dashboard

Enriched Docker image

Enriched Docker image

Codefresh is using this information to fill the deployment history in the GitOps dashboard.

Creating a basic CD pipeline for GitOps

To create a CD pipeline in Codefresh that is responsible for GitOps deployments you must first disable the auto-sync behavior of ArgoCD. You can disable auto-sync either from the GUI or via the command line:

Basic CD pipeline

Basic CD pipeline

With the auto-sync behavior disabled, all Git pushes that happen on the GitOps repo will be ignored by ArgoCD (however ArgoCD will still mark your application as out-of-sync).

You can now create a new pipeline in Codefresh using a standard Git trigger that will monitor the GitOps repository for updates. This way Codefresh is responsible for the GitOps process instead of Argo.

Basic CD pipeline

Basic CD pipeline

The big advantage here is that you can construct a full pipeline over the sync process with multiple steps before or after the sync. For example you could run some smoke tests after the deployment takes place. Here is an example pipeline:

codefresh.yml

version: "1.0"
stages:
  - "pre sync"
  - "sync app"
  - "post sync"
steps:
  pre_sync:
    title: "Pre sync commands"
    type: "freestyle" # Run any command
    image: "alpine:3.9" # The image in which command will be executed
    commands:
      - echo "Sending a metrics marker"
    stage: "pre sync"
  sync_and_wait:
    title: Sync ArgoCD app and wait
    type: argocd-sync
    arguments:
      context: "argo-cd"
      app_name: "${{ARGOCD_APP_NAME}}"
      wait_healthy: true   
    stage: "sync app"
  post_sync:
    title: "Post sync commands"
    type: "freestyle" # Run any command
    image: "alpine:3.9" # The image in which command will be executed
    commands:
      - echo "running smoke tests"
    stage: "post sync"

The pipeline is using the argo-sync plugin that can be used by Codefresh to start the sync process of an application from the Git repo to the cluster.

The name of the context parameter should be the same name you used for your ArgoCD integration.

Using the Argo integration name as a context

Using the Argo integration name as a context

The name of the application should be the same name as the ArgoCD Application.

Argo Application name

Argo Application name

You can use pipeline variables or any other familiar Codefresh mechanism such as shared configuration.

Once the pipeline has finished running the sync status will updated in your GitOps dashboard to reflect the current state.

Working with the GitOps dashboard

After you create an ArgoCD application, you can click on it in the GitOps environment overview and see the respective GitOps screen.

GitOps Dashboard

GitOps Dashboard

This dashboard is the central place for monitoring your application and contains the following information:

  1. Current health and sync status
  2. Deployment graph that shows successful/failed deployments on the selected time period
  3. Complete history of deployments according to Git hash. For each deployment you can also see which Pull Request was used for the commit, who was the committer and which JIRA issues this Pull request is solving (provided that the image was built by a Codefresh pipeline)
  4. The Kubernetes services that belong to this application (on the services tab)

The deployment status is fetched from your ArgoCD integration in a live manner. If at any point, the deployment is not synced with GIT, you will instantly see the out-of-sync status:

Out of sync status

Out of sync status

For each Git hash Codefresh associates the respective Pull Request and Jira issue(s) that affected deployment. To achieve this correlation, Codefresh is enriching the Docker image(s) of the service during the CI process.

You can manually create these annotations with the standard Codefresh annotation support or via the built-in pipeline steps that we will see in the next section.

Filtering the deployment history

You can add filters on the deployment history by using the multi-select field on the top left of the screen.

Filtering options

Filtering options

You can add filters for:

  • Git committer(s)
  • Pull Request number(s)
  • Jira issue(s)

If you define multiple options they work in an OR manner.

Searching the deployment history

For advanced filtering options, the search field on the top right allows you to view only the subset of deployments that match your custom criteria.

Apart from direct text search, the text field also supports a simple query language with the following keywords:

  • issues
  • issue
  • prs
  • pr
  • committer
  • committers
  • service
  • services
  • image
  • images
  • status
  • statuses

The following characters serve as delimiters

  • : define the value for a keyword
  • , define multiple values for a single keyword
  • ; define multiple criteria

Searching deployment history

Searching deployment history

Some examples are:

  • pr:2 - filter the deployment history to show only a specific Pull request
  • issues: SAAS-2111, SAAS-2222 - show only specific issues
  • issue: SAAS-2111; pr:3 ; service: my-app - searching for multiple criteria in OR behavior

Using the search field allows you to quickly find a specific Git commit in the history of the application (and even rollback the deployment as explained in the next sections).

Rolling back Git versions

In the GitOps dashboard you will also see a complete history of all past deployments as recorded in Git. You can select any of the previous version and rollback your application to the respective version.

Rolling back to a previous version

Rolling back to a previous version

The Rollback simply informs the cluster to use a different git hash for the sync process. It doesn’t affect your Git repository and ArgoCD will now show your application as out-of-sync (because the last Git commit no longer matches the status of the cluster).

This rollback behavior is best used as an emergency measure after a failed deployment where you want to bring the cluster back to a previous state in a temporary manner. If you wish to keep the current rollback statue as a permanent status it is best to use the standard git reset/revert commands and change the GitOps repository to its desired state.

Performing automatic Git commits

Usually the Pull Requests that take part in a GitOps workflow are created and approved in a manual way (after code review). You have the option however to fully automate the whole process and rather than opening a Pull Request on both the application repository and the manifest repository, commit automatically the manifest changes inside the pipeline that creates the artifact.

Here is an example pipeline that creates a Docker image and also commits a version change in the Kubernetes manifest to denote the new Docker tag of the application:

Pipeline that commits manifests

Pipeline that commits manifests

There are many ways to change a Kubernetes manifest in a programmatic way, and for brevity reasons we use the yq command line tool.

codefresh.yml

version: "1.0"
stages:
  - "clone"
  - "build"
  - "metadata"
  - "gitops"

steps:
  clone:
    title: "Cloning repository"
    type: "git-clone"
    repo: "my-github-username//gitops-app-source-code"
    revision: '${{CF_REVISION}}'
    stage: "clone"

  build:
    title: "Building Docker image"
    type: "build"
    image_name: "kostiscodefresh/simple-web-app"
    working_directory: "${{clone}}"
    tags:
    - "latest"
    - '${{CF_SHORT_REVISION}}'
    dockerfile: "Dockerfile"
    stage: "build"
    registry: dockerhub  
  enrich-image:
    title: Add PR info
    type: image-enricher
    stage: "metadata"
    arguments:
      IMAGE:  docker.io/kostiscodefresh/simple-web-app:${{CF_SHORT_REVISION}}
      BRANCH: '${{CF_BRANCH}}'
      REPO: 'kostis-codefresh/simple-web-app'
      GIT_PROVIDER_NAME: github-1
  jira-issue-extractor:
    title: Enrich image with jira issues
    type: jira-issue-extractor
    stage: "metadata"
    fail_fast: false
    arguments:
      IMAGE: docker.io/kostiscodefresh/simple-web-app:${{CF_SHORT_REVISION}}
      JIRA_PROJECT_PREFIX: 'SAAS'
      MESSAGE: SAAS-8842
      JIRA_HOST: codefresh-io.atlassian.net
      JIRA_EMAIL: [email protected]
      JIRA_API_TOKEN: '${{JIRA_TOKEN}}'
  clone_gitops:
    title: cloning gitops repo
    type: git-clone
    arguments:
      repo: 'my-github-username//gitops-kubernetes-configuration'
      revision: 'master'
    stage: "gitops"
    when:
      branch:
        only:
          - master    
  change_manifest:
    title: "Update k8s manifest"
    image: "mikefarah/yq" # The image in which command will be executed
    commands:
      - yq w -i deployment.yml spec.template.spec.containers[0].image docker.io/kostiscodefresh/simple-web-app:${{CF_SHORT_REVISION}}
      - cat deployment.yml
    working_directory: "${{clone_gitops}}" 
    stage: "gitops"
    when:
      branch:
        only:
          - master 
  commit_and_push:
    title: Commit manifest
    type: git-commit
    stage: "gitops"
    arguments:
      repo: 'my-github-username//gitops-kubernetes-configuration'
      git: github-1
      working_directory: '/codefresh/volume/gitops-kubernetes-configuration'
      commit_message: Updated manifest
      git_user_name: ${{CF_COMMIT_AUTHOR}}
      git_user_email: ${{CF_COMMIT_AUTHOR}}@acme-inc.com
    when:
      branch:
        only:
          - master      

This pipeline:

  1. Checks out the Git repository that contains the source code
  2. Builds a Docker image and tags it with the Git hash
  3. Enriches the image with the Pull request and ticket information as explained in the previous sections
  4. Checks out the Git repository that contains the Kubernetes manifests
  5. Performs a text replacement on the manifest updating the containers segment with the new Docker image
  6. Commits the change back using the Git commit plugin to the Git repository that contains the manifests.

The CD pipeline (described in the previous section) will detect that commit and use the sync plugin to instruct ArgoCD to deploy the new tag. Alternatively you can setup the ArgoCD project to auto-sync on its own if it detects changes in the Git repository with the manifests.

Using a Git repository for the pipelines

Remember that according to GitOps we should place all application resources on Git. This means that the pipelines themselves must also be present in a Git repository and any change on them should pass from source control.

Even though Codefresh has a powerful inline editor for editing pipelines, as soon as you finish with your pipelines you should commit them in Git and load them from the repository.

Loading a pipeline from GIT

Loading a pipeline from GIT

Once the pipeline is in Git, you should switch the online editor to load the pipeline from the repository instead of the inline text.