Learn how to deploy with Codefresh and ArgoCD
Apart from traditional push-based Helm deployments, Codefresh can also be used for GitOps deployments.
What is GitOps
GitOps is the practice of performing Operations via Git only. The main principles of GitOps are the following:
- The state of the system/application is always stored in Git.
- Git is always the source of truth for what happens in the system.
- If you want to change the state of the system you need to perform a Git operation such as creating a commit or opening a pull request. Deployments, tests, and rollbacks controlled through git flow.
- Once the Git state is changed, then the cluster (or whatever your deployment target is) state should match what is described in the Git repository.
- No hand rolled deployments, no ad-hoc cluster changes, no live configuration changes are allowed. If a change needs to happen, it must be committed to Git first.
GitOps deployments have several advantages compared to traditional imperative deployments. The main one is that the Git repo represents the state of the system, and Git history is essentially the same thing as deployment history. Rollbacks are very easy to perform by simply using a previous Git hash.
Even though GitOps is not specific to Kubernetes, current GitOps tools work great with Kubernetes in the form of cluster controllers. The GitOps controller monitors the state of the Git repository and when a commit happens, the cluster is instructed to match the same state.
Codefresh has native support for GitOps including a graphical dashboard for handling your GitOps deployments:
This guide will explain how you can use GitOps for your own applications.
Setting up your Git Repositories
One of the central ideas around GitOps is the usage of Git for ALL project resources. Even though developers are familiar with using Git for the source code of the application, adopting GitOps means that you need to store in Git every other resource of the application (and not just the source code).
In the case of Kubernetes, this means that all Kubernetes manifests should be stored in a Git repository as well. In the most simple scenario you have the main repository of your application (this is mostly interesting to developers) and a second Git repository with Kubernetes manifests (this is more relevant to operators/SREs).
As a running example you can use:
- The https://github.com/codefresh-contrib/gitops-app-source-code repository for the application code
- The https://github.com/codefresh-contrib/gitops-kubernetes-configuration repository for the Kubernetes configuration
- The https://github.com/codefresh-contrib/gitops-pipelines repository that holds the pipelines
The application code repository contains the source code plus a dockerfile. You can use any Git workflow for this repository. We will set a pipeline in Codefresh that creates a container image on each commit.
The configuration repository holds the kubernetes manifests. This is one of the critical points of GitOps
- The configuration repository holds the manifests that are also present in the Kubernetes cluster
- Every time a commit happens to the configuration repository the cluster will be notified to deploy the new version of the files (we will setup a pipeline for this)
- Every subsequent configuration change should become a Git commit. Ad-hoc changes to the cluster (i.e. with
kubectlcommands) are NOT allowed
We also have a third Git repository for pipelines, because pipelines are also part of the application.
Before continuing fork all 3 repositories in your own GitHub account if don’t have already your own example application.
Connecting ArgoCD and Codefresh
GitOps deployments are powered by ArgoCD so you need an active ArgoCD installation in your cluster to take advantage of the GitOps dashboard in Codefresh.
Follow the instructions for connecting ArgoCD to Codefresh and creating an ArgoCD application
The options are:
- Name - User defined name of the Codefresh environment dashboard
- Project - A way to group/secure applications. Choose default if you have only one project in ArgoCD.
- Application - name of application
- Manual/automatic sync - If automatic when a git commit happens, a deployment will automatically take place.
- Use schema - Kubernetes manifests will be checked for correctness before deployed to the cluster
- source repository - Git repository that holds your Kubernetes manifests
- revision - Revision to be checked out when a deployment happens
- path - folder inside the Git repository that should be searched for manifests (if your Git repo has multiple applications). Use
./if all your manifests are in the root folder.
- cluster - Kubernetes cluster when deployment will take place
- namespace - Kubernetes namespace where the application will be deployed to
- directory recurse - whether to check all folders in the Git repository for manifests in a recursive way.
For a sample application you can use the https://github.com/codefresh-contrib/gitops-kubernetes-configuration repository. Fork the project in your own GitHub account and use that link in the Source repository section.
Once you connect your application you will see it under in the GitOps application screen in the Codefresh UI.
Creating a basic CI Pipeline for GitOps
To take advantage of the GitOps dashboard facilities you also need to setup the correlation between the Docker image and the Pull Requests/issues associated with it. This correlation happens via annotations. The easiest way to annotate your image is by using the pipeline plugins offered by Codefresh for this purpose. Currently we offer the following plugins:
Here is an example Pipeline definition:
- Checks out the source code of an application with the git-clone step
- Builds a docker image
- Annotates the Docker image with the Pull Request information provided by Github
- Annotates the Docker image with a specific Jira issue ticket
You can see the associated metadata in your Docker image dashboard
Codefresh is using this information to fill the deployment history in the GitOps dashboard.
Creating a basic CD Pipeline for GitOps
To create a CD pipeline in Codefresh that is responsible for GitOps deployments you must first disable the auto-sync behavior of ArgoCD. You can disable auto-sync either from the GUI or via the command line:
With the auto-sync behavior disabled, all Git pushes that happen on the GitOps repo will be ignored by ArgoCD (however ArgoCD will still mark your application as out-of-sync).
You can now create a new pipeline in Codefresh using a standard Git trigger that will monitor the GitOps repository for updates. This way Codefresh is responsible for the GitOps process instead of Argo.
The big advantage here is that you can construct a full pipeline over the sync process with multiple steps before or after the sync. For example you could run some smoke tests after the deployment takes place. Here is an example pipeline:
The pipeline is using the argo-sync plugin that can be used by Codefresh to start the sync process of an application from the Git repo to the cluster.
The name of the
context parameter should be the same name you used for your ArgoCD integration.
The name of the application should be the same name as the ArgoCD Application.
You can use pipeline variables or any other familiar Codefresh mechanism such as shared configuration.
Once the pipeline has finished running the sync status will updated in your GitOps dashboard to reflect the current state.
Working with the GitOps Dashboard
After you create an ArgoCD application, you can click on it in the GitOps environment overview and see the respective GitOps screen.
This dashboard is the central place for monitoring your application and contains the following information:
- Current health and sync status
- Deployment graph that shows successful/failed deployments on the selected time period
- Complete history of deployments according to Git hash. For each deployment you can also see which Pull Request was used for the commit, who was the committer and which JIRA issues this Pull request is solving (provided that the image was built by a Codefresh pipeline)
- The Kubernetes services that belong to this application (on the services tab)
- What services and replicas were updated with each deployment.
The deployment status is fetched from your ArgoCD integration in a live manner. If, at any point, the deployment is not synced with GIT, you will instantly see the out-of-sync status. You will get the number of resources that are out of sync. When you click the out-of-sync status, you will get a list of all resources in that status.
For each Git hash Codefresh associates the respective Pull Request and Jira issue(s) that affected deployment. To achieve this correlation, Codefresh is enriching the Docker image(s) of the service during the CI process.
You can manually create these annotations with the standard Codefresh annotation support or via the built-in pipeline steps that we will see in the next section.
You can find helpful tips if you hover your mouse on the PR number, the issue, the Git commiter and so on.
For each deployment you can also see a before/after view of the pods/replicas that were affected.
Filtering the Deployment History
You can add filters on the deployment history by using the multi-select field on the top left of the screen.
You can add filters for:
- Git committer(s)
- Pull Request number(s)
- Jira issue(s)
If you define multiple options they work in an OR manner.
Searching the Deployment History
For advanced filtering options, the search field on the top right allows you to view only the subset of deployments that match your custom criteria.
Apart from direct text search, the text field also supports a simple query language with the following keywords:
The following characters serve as delimiters
:define the value for a keyword
,define multiple values for a single keyword
;define multiple criteria
Some examples are:
pr:2- filter the deployment history to show only a specific Pull request
issues: SAAS-2111, SAAS-2222- show only specific issues
issue: SAAS-2111; pr:3 ; service: my-app- searching for multiple criteria in OR behavior
Using the search field allows you to quickly find a specific Git commit in the history of the application (and even rollback the deployment as explained in the next sections).
Current State of Application
The current state tab shows a hierarchical view of your cluster resource for your application.
At the top of the screen you have several filters available:
- Kind - choose a specific type of Kubernetes resource
- Health - status of the resource
- Sync state - GitOps status of the resource
- Free search - search any resource by name
Tagging GitOps Application
- Navigate to the GitOps dashboard.
- To the application’s right (next to the Health Column), click the three dots to open the More Action Dropdown.
- Select Add/Edit Tags.
- Click the +tags to add tags.
- Alternatively, click the “x” next to the tag to remove it.
- Click Save.
Rolling Back Git Versions
In the GitOps dashboard you will also see a complete history of all past deployments as recorded in Git. You can select any of the previous versions and rollback your application to the respective version.
The Rollback simply informs the cluster to use a different git hash for the sync process. It doesn’t affect your Git repository and ArgoCD will now show your application as out-of-sync (because the last Git commit no longer matches the status of the cluster).
This rollback behavior is best used as an emergency measure after a failed deployment where you want to bring the cluster back to a previous state in a temporary manner. If you wish to keep the current rollback status as a permanent status it is best to use the standard
git reset/revert commands and change the GitOps repository to its desired state.
Gitops ABAC Support For Rollback Action
- Go to Account Settings > Permissions > Teams Tab > Gitops.
- Select the Team.
- Chose what the Team can do and click apply.
- Select the tags of the applications and click apply.
- Click Add Rule when done.
Performing Automatic Git Commits
Usually the Pull Requests that take part in a GitOps workflow are created and approved in a manual way (after code review). You have the option however to fully automate the whole process and rather than opening a Pull Request on both the application repository and the manifest repository, commit automatically the manifest changes inside the pipeline that creates the artifact.
Here is an example pipeline that creates a Docker image and also commits a version change in the Kubernetes manifest to denote the new Docker tag of the application:
There are many ways to change a Kubernetes manifest in a programmatic way, and for brevity reasons we use the yq command line tool.
- Checks out the Git repository that contains the source code
- Builds a Docker image and tags it with the Git hash
- Enriches the image with the Pull request and ticket information as explained in the previous sections
- Checks out the Git repository that contains the Kubernetes manifests
- Performs a text replacement on the manifest updating the
containerssegment with the new Docker image
- Commits the change back using the Git commit plugin to the Git repository that contains the manifests.
The CD pipeline (described in the previous section) will detect that commit and use the sync plugin to instruct ArgoCD to deploy the new tag. Alternatively you can setup the ArgoCD project to auto-sync on its own if it detects changes in the Git repository with the manifests.
Using the App-of-Apps pattern
The GitOps dashboard has native support for the app-of-apps pattern. If you have a number of applications that are related and you always install them as a set in your cluster you can group them in a single Application. The parent application can be defined using declarative Argo Resources.
As an example, you might find that you always install in your cluster Linkerd, Prometheus and Ambassador. You can group all of them in a single Application and deploy them all at once.
You can find an existing example of app-of-apps at https://github.com/argoproj/argocd-example-apps/tree/master/apps. It is using Helm, but you can use any other Kubernetes templating mechanism such as Kustomize (or even plain manifests).
Once you deploy the application with Codefresh, you will see the parent app in the dashboard with a small arrow:
You can expand the application by clicking on the arrow to inspect its child applications.
Then you can either click on the parent application or any of the children to visit the respective dashboard. In the dashboard of the parent application, you will also be notified for its components after each deployment under the “Updated Applications” header:
Note that the app of apps pattern is best used for related but not interdependent applications. If you have applications that depend on each other (e.g. frontend that needs backend and backend that needs a DB) we suggest you use the standard Helm dependency mechanism.
Integrating Codefresh and Jira
Note that Codefresh currently has to provide you with access to use the Jira Marketplace App. Please get in touch for more information.
Setting up the Codefresh Jira integration provides
- Higher observability of deployments within your GitOps Dashboard
- Higher observability of deployments within your Jira Account
Our integration section provides further details on ways to set-up the connection.
Once set-up, you will be able to view information from Jira in the Codefresh GitOps Dashboard. Additionally, Jira will display
- The build status across environments
- The deployment history
- Tickets and how they correlate to deployments
The following screenshots show examples of the provided information. Here is the deployments details for a ticket in JIRA:
And here is a complete timeline of your deployments and the feature they contain.
Using a Git repository for the pipelines
Remember that according to GitOps we should place all application resources on Git. This means that the pipelines themselves must also be present in a Git repository and any change on them should pass from source control.
Once the pipeline is in Git, you should switch the online editor to load the pipeline from the repository instead of the inline text.