Updated August 2024
If you are using Argo CD, you may be already familiar with how the Application CRD (Custom Resource Definition) object helps you logically group together your Kubernetes Manifests. The Application object is the atomic unit of work in Argo CD, and you should think of all your Kubernetes objects that are in an Application as a single entity.
Applications are also autonomous. Meaning that, by design, one Application doesn’t know about the status or health of another Application. This could pose a challenge in organizations where they are implementing a microservices architecture, with each component being in its own Application CR (Custom Resource). Some examples include:
- Database -> Backend
- Queue -> Queue workers -> Backend
- Kyverno/Opa -> Apps that need to be limited
- Database -> Backend -> Frontend
Since there is no way, currently, to set up Application dependencies natively; is there a way to do it with what’s available?
The answer is: Yes, by combining App-of-Apps and Syncwaves.
App-of-Apps Pattern
The App-of-Apps pattern was a design that came from the community of Argo CD users. The App-of-Apps design is basically an Argo CD Application made up of other Argo CD Applications. Initially, the use case was for bootstrapping. Administrators needed a way to deploy Argo CD Applications using Argo CD itself. The natural fit was to create an Application made up of other Argo CD Applications (since an Application is just another Kubernetes object).
Given the above example, we’d have the following Argo CD Applications:
- Cert-Manager
- Backend Application
- Caching System
- Kyverno
- Frontend Application
- Ingress
Instead of deploying 6 individual Argo CD Application, you can deploy one Argo CD Application that deploys the other 6 Applications for you.
This also provided a way to bootstrap a cluster with your applications with a convenient “entry point”; It was also a way of having a “watcher” Application. Another benefit was that you could use other Argo CD paradigms in this App-of-Apps pattern. One of which is to use SyncWaves
SyncWaves
Syncwaves and Synchooks are a way to order how Argo CD applies individual manifests within an Argo CD Application. The order is done by annotating the object with the order you’d like to apply the manifest, number based with lowest going first (negatives are allowed). For example, if you have your Deployment as “0” and your Service as “1” – Argo CD will apply the Deployment first, wait for it to report back healthy, then apply the Service.
You may already have used Syncwaves and App-of-Apps on their own. But what happens if we use them together? Can we order with sync waves individual Argo CD Applications instead of just Kubernetes resources?
Prerequisites
You need to set up a few things before you can integrate SyncWaves with your App-of-Apps deployment and create Argo CD Application dependencies..
Argo CD Application Health
Argo CD has health checks for several standard Kubernetes objects built-in. These checks then are bubbled up to the overall Application health status as one unit. For example, an Application that has a Service and a Deployment will be marked “healthy” only if both objects are considered healthy.
Some of the built-in health checks include (but are not limited to): Deployment, ReplicaSet, StatefulSet DaemonSet, Service, Ingress, and PersistentVolumeClaim. There is also the ability to add custom health checks as well. You can read more about it in the official documentation.
Also described in the official documentation, you need to tell Argo CD how to check for the Application Custom Resource overall health. This can be done by modifying the argocd-cm ConfigMap and adding a resource customization. For example
apiVersion: v1 --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd labels: app.kubernetes.io/name: argocd-cm app.kubernetes.io/part-of: argocd data: resource.customizations: | argoproj.io/Application: health.lua: | hs = {} hs.status = "Progressing" hs.message = "" if obj.status ~= nil then if obj.status.health ~= nil then hs.status = obj.status.health.status if obj.status.health.message ~= nil then hs.message = obj.status.health.message end end end return hs
This ensures that the Application controller reports the health of the Application CR correctly. Starting in Argo CD version 1.8, you must put this setting in (see issue 3781 for more details).
Readiness/Liveness Probes
Another part of getting Application dependencies up and running with App-of-Apps, is to make sure that your deployments/statefulsets/daemonsets have the proper readiness/liveness probes set up. This is important because Argo CD will look at the health of the object and use that to determine if the Application is healthy. This can cause issues if you don’t have proper readiness/liveness probes.
For example, a Deployment without readiness/liveness probes will be marked healthy as soon as the Deployment is applied and the container is running. This will cause the Application to be marked as healthy, when in fact, your application may take some time to come up. In order to get your Application dependency working optimally, you’ll need to set these. Here is a snippet of a Deployment manifest that has these set for a web application.
livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 3 readinessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 3
The above snippet shows that the container will be marked as “alive” when it returns the expected HTTP response when probing that port and path; Similarly the container will be marked “ready” when that same probe returns the expected HTTP response.
Example
I will be going over an example using a repo called application-dependency, which is a 3 tiered application made up of a frontend app, backend api, and a database. These three apps are deployed, individually, using an Argo CD Application for each of them.
I want them to deploy in a specific order, and the logical way is to have the database come up first, then the backend app, then the frontend. Here is a simple diagram:
In order to get this functionality, I need to do the following:
- Create the individual Argo CD Applications for each tier (already done here)
- Create the Parent Argo CD Application that deploys them (already done here)
- Update Argo CD with the health check for the Applications
- Make sure I have proper probes setup for my objects
- Annotate my Applications with the right syncwave
In order to add the health check for Argo CD Applications, I created a patch file; it should look something like this:
$ cat patch-argocd-cm.yaml data: resource.customizations.health.argoproj.io_Application: | hs = {} hs.status = "Progressing" hs.message = "" if obj.status ~= nil then if obj.status.health ~= nil then hs.status = obj.status.health.status if obj.status.health.message ~= nil then hs.message = obj.status.health.message end end end return hs
Then update the argocd-cm ConfigMap:
$ kubectl patch cm/argocd-cm --type=merge -n argocd --patch-file patch-argocd-cm.yaml
NOTE : This is just an example for this blog. Ideally you would use GitOps to manage this Argo CD ConfigMap. Also this can be done automatically with ArgoCD Autopilot.
You can verify with the following command:
$ kubectl get cm argocd-cm -n argocd -o yaml
Now that the argocd-cm ConfigMap has been updated, you can annotate the Argo CD Application definition with the syncwave number. First, I annotate the Database Application with “1”, as I want it to get deployed first. You can see the full manifest for the database in my repo, here is a snippet of the YAML:
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: argocd.argoproj.io/sync-wave: "1" name: my-db namespace: argocd
Next, I annotate my backend API Argo CD Application with “2”, since I want it to come up only after the Database Application is finished. It’s a similar manifest, so here is a snippet.
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: argocd.argoproj.io/sync-wave: "2" name: my-api namespace: argocd
Now finally, I have my frontend application manifest, which has a syncwave annotation of “3”, here is a snippet of this file:
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: argocd.argoproj.io/sync-wave: "3" name: my-frontend namespace: argocd
Before I go on to the App-of-App setting, I have to point out that all 3 of these Applications have readiness/liveness probes set up. For example the Frontend Deployment and the API Deployment have them set.
Now that we have the Argo CD Application health check set up, annotated the Applications in the order I want them in, and we made sure we had the readiness/liveness probes in our apps; we can now take a look at the “Parent” Argo CD Application in this App-of-Apps configuration.
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: all-apps-in-order namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: source: path: apps repoURL: 'https://github.com/kostis-codefresh/application-dependency-argocd.git' targetRevision: main destination: namespace: argocd server: 'https://kubernetes.default.svc' project: default syncPolicy: automated: prune: true selfHeal: true retry: limit: 5 backoff: duration: 5s maxDuration: 3m0s factor: 2 syncOptions: - CreateNamespace=true
There is nothing inherently special about this Application (as in it just looks like a “regular” application), and that’s because there isn’t! The thing to note is that apps is the path in my repo where the other Argo CD Applications are.
So far we have:
- Created the Argo CD Applications for my apps
- Created the Parent Argo CD Application that will deploy those Applications
- Added the proper Argo CD Application health check to Argo CD
- Made sure my Deployments had the right readiness/liveness probes
- Annotated my Argo CD Applications with the Syncwave I wanted
Now, let’s see this in action! Now that those things are in place, I can apply the “parent” Argo CD Application.
$ kubectl apply -f all-apps.yml
Here you’ll see two Applications in the Argo CD UI, the “parent” Application and the first Application in the syncwave, in my case: the database Application.
After that completes, the next Application in the syncwave starts; in this case, it’s the API Application.
Finally, the last Application in the wave will start to deploy, which is the frontend Application.
The App-of-Apps is now complete! Once all 3 Applications are synced and healthy, the “Parent” Application will report as synced and healthy.
ApplicationSets
ApplicationSets are an evolution of what the App-of-Apps pattern provides. It is meant to not only help with bootstrapping but also with templating out an Argo CD Application. From a high level, an ApplicationSet has the ability to use a single manifest to target multiple Kubernetes clusters. Furthermore, to use a single manifest to deploy multiple applications from one or more Git repos.
With all that ApplicationSets give you, there is one caveat. There is no way to set up Application dependencies with ApplicationSets. It’s something that has been proposed (including using DAG), and you can track that upstream; so you will still need to use App-of-Apps if you need to set up Application dependencies.
Conclusion
In this blog, we talked about how some organizations using Argo CD use an Application per microservice. We’ve talked about the challenges with that design and explored how you can mitigate those challenges by using App-of-Apps along with Syncwaves. Finally we touched on ApplicationSets and how you might still need App-of-Apps in the short term.
Like this blog and want to learn more tips and tricks about Argo CD and GitOps? Get certified by taking our certification course on GitOps and Argo CD! Join today at https://codefresh.io/get-certified