Argo CD Application Dependencies

Argo CD Application Dependencies

8 min read

Updated 4/24/2023

If you are using Argo CD, you may be already familiar with how the Application CRD (Custom Resource Definition) object helps you logically group together your Kubernetes Manifests. The Application object is the atomic unit of work in Argo CD, and you should think of all your Kubernetes objects that are in an Application as a single entity.

Applications are also autonomous. Meaning that, by design, one Application doesn’t know about the status or health of another Application. This could pose a challenge in organizations where they are implementing a microservices architecture, with each component being in its own Application CR (Custom Resource). Some examples include:

  1. Database -> Backend
  2. Queue -> Queue workers -> Backend
  3. Kyverno/Opa -> Apps that need to be limited
  4. Database -> Backend -> Frontend

Since there is no way, currently, to set up Application dependencies natively; is there a way to do it with what’s available?

The answer is: Yes, by combining App-of-Apps and Syncwaves.

App-of-Apps Pattern

The App-of-Apps pattern was a design that came from the community of Argo CD users. The App-of-Apps design is basically an Argo CD Application made up of other Argo CD Applications. Initially, the use case was for bootstrapping. Administrators needed a way to deploy Argo CD Applications using Argo CD itself. The natural fit was to create an Application made up of other Argo CD Applications (since an Application is just another Kubernetes object).

Given the above example, we’d have the following Argo CD Applications:

  • Cert-Manager
  • Backend Application
  • Caching System
  • Kyverno
  • Frontend Application
  • Ingress

Instead of deploying 6 individual Argo CD Application, you can deploy one Argo CD Application that deploys the other 6 Applications for you.

This also provided a way to bootstrap a cluster with your applications with a convenient “entry point”; It was also a way of having a “watcher” Application. Another benefit was that you could use other Argo CD paradigms in this App-of-Apps pattern. One of which is to use SyncWaves

SyncWaves

Syncwaves and Synchooks are a way to order how Argo CD applies individual manifests within an Argo CD Application. The order is done by annotating the object with the order you’d like to apply the manifest, number based with lowest going first (negatives are allowed). For example, if you have your Deployment as “0” and your Service as “1” – Argo CD will apply the Deployment first, wait for it to report back healthy, then apply the Service.

You may already have used Syncwaves and App-of-Apps on their own. But what happens if we use them together? Can we order with sync waves individual Argo CD Applications instead of just Kubernetes resources?

Prerequisites

You need to set up a few things before you can integrate SyncWaves with your App-of-Apps deployment and create Argo CD Application dependencies..

Argo CD Application Health

Argo CD has health checks for several standard Kubernetes objects built-in. These checks then are bubbled up to the overall Application health status as one unit. For example, an Application that has a Service and a Deployment will be marked “healthy” only if both objects are considered healthy.

Some of the built-in health checks include (but are not limited to): Deployment, ReplicaSet, StatefulSet DaemonSet, Service, Ingress, and PersistentVolumeClaim. There is also the ability to add custom health checks as well. You can read more about it in the official documentation.

Also described in the official documentation, you need to tell Argo CD how to check for the Application Custom Resource overall health. This can be done by modifying the argocd-cm ConfigMap and adding a resource customization. For example

apiVersion: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations: |
    argoproj.io/Application:
      health.lua: |
        hs = {}
        hs.status = "Progressing"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            if obj.status.health.message ~= nil then
              hs.message = obj.status.health.message
            end
          end
        end
        return hs

This ensures that the Application controller reports the health of the Application CR correctly. Starting in Argo CD version 1.8, you must put this setting in (see issue 3781 for more details).

Readiness/Liveness Probes

Another part of getting Application dependencies up and running with App-of-Apps, is to make sure that your deployments/statefulsets/daemonsets have the proper readiness/liveness probes set up. This is important because Argo CD will look at the health of the object and use that to determine if the Application is healthy. This can cause issues if you don’t have proper readiness/liveness probes.

For example, a Deployment without readiness/liveness probes will be marked healthy as soon as the Deployment is applied and the container is running. This will cause the Application to be marked as healthy, when in fact, your application may take some time to come up. In order to get your Application dependency working optimally, you’ll need to set these. Here is a snippet of a Deployment manifest that has these set for a web application.

livenessProbe:
  httpGet:
    path: /
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 3
readinessProbe:
  httpGet:
    path: /
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 3

The above snippet shows that the container will be marked as “alive” when it returns the expected HTTP response when probing that port and path; Similarly the container will be marked “ready” when that same probe returns the expected HTTP response.

Example

I will be going over an example using my repo called golist, which is a 3 tiered application made up of a frontend app, backend api, and a database. These three apps are deployed, individually, using an Argo CD Application for each of them.

I want them to deploy in a specific order, and the logical way is to have the database come up first, then the backend app, then the frontend. Here is a simple diagram:

In order to get this functionality, I need to do the following:

  1. Create the individual Argo CD Applications for each tier (already done here)
  2. Create the Parent Argo CD Application that deploys them (already done here)
  3. Update Argo CD with the health check for the Applications
  4. Make sure I have proper probes setup for my objects
  5. Annotate my Applications with the right syncwave

In order to add the health check for Argo CD Applications, I created a patch file; it should look something like this:

$ cat patch-argocd-cm.yaml
data:
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs

Then update the argocd-cm ConfigMap:

$ kubectl patch cm/argocd-cm --type=merge -n argocd --patch-file patch-argocd-cm.yaml

NOTE : This is just an example for this blog. Ideally you would use GitOps to manage this Argo CD ConfigMap. Also this can be done automatically with ArgoCD Autopilot.

You can verify with the following command:

$ kubectl get cm argocd-cm -n argocd -o yaml

Now that the argocd-cm ConfigMap has been updated, you can annotate the Argo CD Application definition with the syncwave number. First, I annotate the Database Application with “1”, as I want it to get deployed first. You can see the full manifest for the database in my repo, here is a snippet of the YAML:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  name: golist-db
  namespace: argocd

Next, I annotate my backend API Argo CD Application with “2”, since I want it to come up only after the Database Application is finished. It’s a similar manifest, so here is a snippet.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  name: golist-api
  namespace: argocd

Now finally, I have my frontend application manifest, which has a syncwave annotation of “3”, here is a snippet of this file:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "3"
  name: golist-frontend
  namespace: argocd

Before I go on to the App-of-App setting, I have to point out that all 3 of these Applications have readiness/liveness probes set up. For example the Frontend Deployment and the API Deployment have them set.

Now that we have the Argo CD Application health check set up, annotated the Applications in the order I want them in, and we made sure we had the readiness/liveness probes in our apps; we can now take a look at the “Parent” Argo CD Application in this App-of-Apps configuration.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: golist
  namespace: argocd
spec:
  source:
    path: argocd/applications
    repoURL: 'https://github.com/christianh814/golist'
    targetRevision: main
  destination:
    namespace: argocd
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m0s
        factor: 2
    syncOptions:
      - CreateNamespace=true

There is nothing inherently special about this Application (as in it just looks like a “regular” application), and that’s because there isn’t! The thing to note is that argocd/application is the path in my repo where the other Argo CD Applications are.

So far we have:

  1. Created the Argo CD Applications for my apps
  2. Created the Parent Argo CD Application that will deploy those Applications
  3. Added the proper Argo CD Application health check to Argo CD
  4. Made sure my Deployments had the right readiness/liveness probes
  5. Annotated my Argo CD Applications with the Syncwave I wanted

Now, let’s see this in action! Now that those things are in place, I can apply the “parent” Argo CD Application.

$ kubectl apply -f golist-app-of-apps.yaml

Here you’ll see two Applications in the Argo CD UI, the “parent” Application and the first Application in the syncwave, in my case: the database Application.

After that completes, the next Application in the syncwave starts; in this case, it’s the API Application.

Finally, the last Application in the wave will start to deploy, which is the frontend Application.

The App-of-Apps is now complete! Once all 3 Applications are synced and healthy, the “Parent” Application will report as synced and healthy.

ApplicationSets

ApplicationSets are an evolution of what the App-of-Apps pattern provides. It is meant to not only help with bootstrapping but also with templating out an Argo CD Application. From a high level, an ApplicationSet has the ability to use a single manifest to target multiple Kubernetes clusters. Furthermore, to use a single manifest to deploy multiple applications from one or more Git repos.

With all that ApplicationSets give you, there is one caveat. There is no way to set up Application dependencies with ApplicationSets. It’s something that has been proposed (including using DAG), and you can track that upstream; so you will still need to use App-of-Apps if you need to set up Application dependencies.

Conclusion

In this blog, we talked about how some organizations using Argo CD use an Application per microservice. We’ve talked about the challenges with that design and explored how you can mitigate those challenges by using App-of-Apps along with Syncwaves. Finally we touched on ApplicationSets and how you might still need App-of-Apps in the short term.

Like this blog and want to learn more tips and tricks about Argo CD and GitOps? Get certified by taking our certification course on GitOps and Argo CD! Join today at https://codefresh.io/get-certified

How useful was this post?

Click on a star to rate it!

Average rating 5 / 5. Vote count: 1

No votes so far! Be the first to rate this post.

5 thoughts on “Argo CD Application Dependencies

  1. Hi Christian, thank you very much for an article. I’d have question about the “sync wave” for the parent ArgoCD Application. It happened to me in past that when I “synced” parent Application while one from leaf Application were or entered in “Progressing” state (e.g. due CrashLooping Pod ), the parent Application entered in ‘waiting for parent application to be healthy” deadlock. I have to say that I don’t have auto-sync enabled and this is really an edge case but I’d be interested whether setting argocd.argoproj.io/sync-wave annotation for parent Application to “4” ( according to your example ) resolves issue I faced in past.

  2. Hi Christian,

    Your information was very helpful. I would like to know if there is a possibility of synchronizing one or more apps collaterally. i,e I want to sync app-A with syncwave -1 and then app-B, app-C with syncwave-2. Would it be possible?

  3. Hello Cristian,

    We have some strange situation – when we want to remove (delete) all applications – deleting parent application will only remove child CRDs but application is still remain deployed on k8s. Did you have some similar situation?

    Thanks in advance,

  4. Hi Christian

    Great article! However, I’m experiencing an issue where the sync wave order of child applications is only obeyed during the initial deployment. Subsequent updates to the applications does not obey the sync wave order. The applications sync in a seemingly random order. I don’t suppose you know how to workaround this? I’ve seen several issues on GitHub stating similar.

Leave a Reply

Your email address will not be published. Required fields are marked *

Comment

Build your GitOps skills and credibility today with a GitOps Certification.

Get GitOps Certified

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments