Using Helm Hierarchies in Multi-Source Argo CD Applications for Promoting to Different GitOps Environments

Using Helm Hierarchies in Multi-Source Argo CD Applications for Promoting to Different GitOps Environments

14 min read

Two of the most popular guides we’ve written are the GitOps promotion guide and the ApplicationSet guide. Used together, they explain an end-to-end solution for organizing your GitOps applications and promoting them between different environments, while keeping things DRY by using application sets. 

Both of these guides use Kustomize. We offered some hints for Helm users, but this is a dedicated guide for Helm applications. Organizations that choose Helm have several other challenges to overcome when adopting GitOps. This is because you can also store Helm charts in dedicated Helm repos (apart from Git). Helm charts have versions on their own.  

Unfortunately, we’ve seen teams try to replicate features in Argo CD that Helm already supports. Or, some teams try to adopt Argo CD without first understanding the basics of Helm.

In this guide, we include several Helm-related topics to consider when adopting GitOps and Argo CD in your organization. 

We describe:

  • The recommended Helm structure for GitOps repositories
  • When to use the multi-source feature of Argo CD and when not to use it
  • How to create Helm value hierarchies and why this is important
  • Common Helm bad practices and misconceptions that people carry over to Argo CD

If you like our guides but wonder how our advice works with Helm and not Kustomize, then this guide is for you!

You can find the full example at https://github.com/kostis-codefresh/multi-sources-example.

How Helm values work

The original design for Helm is to have a generic chart (without any hardcoded configuration) and then use several different value files that hold environment-specific configuration. We’ve seen this scenario in detail in our guide about Helm values. It’s also possible to have default values for a chart if no external values are added.

You need to decide if your Helm chart will map to a specific application or not. One approach is to have a Helm chart for every application you develop. In this case, the Helm chart has 2 versions. One version keeps a history about the chart itself and the second version (appVersion in the Chart.yml file) keeps track of the application contained in the chart.

With the second approach, you have a reusable “single” chart that’s targeted at many applications. It doesn’t hold a specific application on its own. You always need to combine it with values that define the image/tag/configuration.

Both approaches are valid and Argo CD supports both. It makes sense, however, to select a single strategy for your organization.

Assuming the classic trinity of QA/Staging/Prod you should be able to easily install any chart with the appropriate configuration, like this:

helm install ./my-chart/ --generate-name -f values-qa.yaml
helm install ./my-chart/ --generate-name -f values-staging.yaml
helm install ./my-chart/ --generate-name -f values-production.yaml

In this very simple example, the charts are saved in Git as normal files along with their possible configurations.

Creating reusable configuration with Helm value hierarchies

The structure explained above works for most organizations. In some cases, however, the application has a complex configuration you need to define for a large number of environments. You could duplicate the values file for each environment, but this would create a cumbersome setup where you need to change common configuration between some environments.

To overcome this challenge, Helm also supports value hierarchies. This means that instead of defining a single value file when installing a chart, you can define multiple. Helm then merges all value files, and for properties defined multiple times, the last file wins.

This means that you can do:

helm install ./my-chart/ --generate-name -f common.yaml -f more.yaml -f some-more.yaml

If you have properties that are both in common.yaml and in more.yaml, the latter will prevail. In the same way, all properties defined in some-more.yaml will function as “overrides” for everything else you defined in the previous files.

The obvious advantage of this pattern is that you can create a hierarchy of Helm value files where “early” files contain common values for most environments and “late” files contain specific values for individual deployments.

A possible setup would be:

common-values.yaml
+-----all-prod-envs.yaml
    +----specific-prod-cluster.yaml

This means common-values get shared between ALL your deployments unless they’re overridden in “all-prod-envs” for all your production environments, which can also be overridden by “specific-prod-cluster.yaml” for a specific cluster.

To make this work, you need to apply all files in the expected order – remembering the last one wins.

helm install ./my-chart/ --generate-name -f common-values.yaml -f all-prod-envs.yaml -f specific-prod-cluster.yaml

If you’re using Helm for your own applications, it’s essential you create an appropriate hierarchy for all your Helm applications that balances flexibility with minimal duplication.

It’s also important to remember that Helm has these hierarchies built-in. They’re unrelated to Argo CD.

Using Helm value hierarchies with Argo CD

After spending time designing your hierarchy of Helm values, you’re now ready to use Argo CD. The good news is that Argo CD supports Helm hierarchies out-of-the-box without any additional configuration. 

Here’s an example application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    chart: my-chart
    repoURL: https://github.com/my-example-app
    targetRevision: 2.43
    helm:
      valueFiles:
      - common-values.yaml
      - all-prod-envs.yaml            
      - specific-prod-cluster.yaml
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default 

The valueFiles property accepts an array of value files. Like the Helm CLI, each file mentioned in the list overrides all values from the previous files. The last file wins over the previous ones.

Why is this important? We’ve seen several teams try to replicate this pattern of default/environment-specific values using several hacks and workarounds, as we described in the previous guide. Usually, the teams that use these workarounds aren’t aware how Helm value hierarchies work.

Don’t fall into the trap of adding another layer of templating to your applications or appsets just to replicate Helm value hierarchies.

Using local Helm values with charts from Helm repositories

All the previous examples assume your Helm chart and its values are in Git. That’s always our recommendation, especially for your own applications. Some organizations, however, prefer to store charts in Helm repositories or OCI registries. Argo CD supports this scenario, but you need to understand the implications:

  • Helm repositories (and OCI artifacts) are mutable. This means that you might download Helm chart 1.2.3 today with different contents than what you downloaded yesterday as version 1.2.3. And, of course, this breaks the GitOps principles
  • There’s no built-in auditing or history for changes made in the chart. If you use Git you get these features for “free”.
  • There are risks regarding integrity and security. Can you verify that the chart you downloaded from the registry is the one the creator uploaded? Again, if you use your own Git repo, you can monitor and optionally approve all changes that happen on the chart.

If you insist on loading charts from external repositories, you should still keep your own values in your own Git repository. Argo CD can then merge the two using multiple sources

Multiple sources is a feature of Argo CD created specifically for this scenario. In the most basic case, you should define 2 sources. One contains your external Helm chart and the other your Helm values.

Like before, you can also have Helm hierarchies.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-fancy-app
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: prod-us
  sources:
    - repoURL: https://acme.github.io/my-charts/
      chart: my-app
      targetRevision: 2.43
      helm:
        valueFiles:
        - $values/common-values.yaml      
        - $values/all-prod-envs.yaml                
        - $values/specific-prod-cluster.yaml
    - repoURL: 'https://github.com/example-org/all-my-values.git'
      targetRevision: HEAD
      ref: values

Notice here the relationship between the “ref:values” field and the “$values” prefix in the helm property. Of course, you can choose any name for the reference and not just “values”.

It should be clear that with this syntax, Argo CD has built-in support for Helm value hierarchies. Argo CD will load the chart from the external source and combine it with all the value files defined in order (last one wins).

On a related note, don’t abuse the multiple source feature of Argo CD. It’s NOT a generic way to group applications. Usually, you should have just 2 sources (as shown below) of related data that form a single application. If you find yourself grouping unrelated apps or having more than 2 to 3 sources, you need to look at app-of-apps and Application Sets instead.

Using Application Sets with Helm value hierarchies

If you deploy your Helm charts as an individual Argo CD application, you should do the same for your Application Sets. We covered Application Sets in detail in our previous guide

Again, the key point here is that Application Sets also support Helm value hierarchies. And, like everything else in the application, you can template your chart values, folder, revision, or anything else you want.

Application Sets also work with multiple sources. So we can combine everything that we have seen so far in a single Application Set, like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-fancy-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: qa
        type: non-prod           
      - env: prod-eu 
        type: prod      
      - env: prod-us
        type: prod      
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://acme.github.io/my-charts/
          chart: my-chart
          targetRevision: 2.43
          helm:
            valueFiles:
            - $values/common-values.yaml          
            - $values/all-{{.type}}-envs.yaml              
            - $values/{{.env}}-values.yaml  
        - repoURL: 'https://github.com/example-org/all-my-values.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

This Application Set defines 3 environments (qa, prod-eu, prod-us). It uses Helm hierarchies and an external chart.

Notice that we can template the location of all value files without any additional templating solution.

We see many teams who place their application CRDs (or applicationSet CRDs) inside Helm charts to template them on a second level. This complexity isn’t needed. Application Sets on their own cover most common scenarios. Application Sets are a way of templating (the same as Helm). Using Helm to generate Application Sets (or Applications) is a recipe for disaster.

If there’s something you can’t template with Application Sets, it’s better to open an enhancement request instead of bringing yet another layer of templating into the mix. 

A complete example of GitOps environments with Helm values

Let’s see how everything works with a semi-realistic example. You can find all the Kubernetes manifests and Argo CD resources at https://github.com/kostis-codefresh/multi-sources-example.

The repository contains:

The environments are similar to those defined in the promotion guide, but this time we use Helm values instead of Kustomize overlays.

We have the following environments:

  • Integration testing (non-GPU and GPU-enabled)
  • QA (only one instance, located in the US)
  • Staging in EU, Asia, US
  • Production only in EU and US

Similar to Kustomize reusable components, we also have a structure of reusable Helm values that you can use in a hierarchical manner.

The expectation is that to form a full Helm release, you need to combine multiple value files in order.

For example, to deploy the application to the “Staging” environment in the EU region, you need to do the following with plain Helm:

helm install ./my-chart/ --generate-name --create-namespace -n staging-eu \
-f my-values/common-values.yaml \
-f my-values/app-version/staging-values.yaml \
-f my-values/env-type/non-prod-values.yaml \
-f my-values/regions/eu-values.yaml \
-f my-values/envs/staging-eu-values.yaml

Remember that the last setting wins. You can always override the common settings with different configurations with files deeper in the tree.
Of course, we don’t want to use Helm, but install the same hierarchy with Argo CD. Here’s the respective Argo CD application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-eu-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io    
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: staging-eu
  sources:
    - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
      path: my-chart
      targetRevision: HEAD
      helm:
        valueFiles:
        - $values/my-values/common-values.yaml          
        - $values/my-values/app-version/staging-values.yaml              
        - $values/my-values/env-type/non-prod-values.yaml  
        - $values/my-values/regions/eu-values.yaml                
        - $values/my-values/envs/staging-eu-values.yaml
    - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
      targetRevision: HEAD
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true  
    automated:
      prune: true
      selfHeal: true

Notice again the hierarchy of value files. The file also follows the best practices outlined in our Application Set guide

If you deploy this application with Argo CD, you can easily verify that the settings are correctly defined according to all the values “merged” in the correct order (the last one wins in case of duplicate properties).

That is great. But as we have said many times before, you shouldn’t deal with individual applications. We can also use Helm hierarchies with Application Sets.

There are many ways to slice and dice your apps. The following examples just show the possible combination of Helm hierarchies.

Application Set on a single dimension

Here’s a very simple Application Set that includes a list generator to model the regions for a single application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-staging-appset
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - region: us
      - region: eu
      - region: asia      
  template:      
    metadata:
      name: 'staging-{{.region}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
          path: my-chart
          targetRevision: HEAD
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/staging-values.yaml              
            - $values/my-values/env-type/non-prod-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/staging-{{.region}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: 'staging-{{.region}}'


      # Sync policy
      syncPolicy:
        syncOptions:
          - CreateNamespace=true  
        automated:
          prune: true
          selfHeal: true

Here, the Helm value locations are templated according to the application’s region. An alternative is to use the list generator for a list of apps, clusters, or environment types.Deploying this application set creates 3 applications, one for each region.

Again, you can easily verify that each deployment gets the appropriate settings according to the respective Helm value hierarchy.

Application Set with multiple templated values

In the previous example, we only looped on the “region” property of the value folders. We can use the list generator to loop over several different properties.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: integration-gpu
        region: us
        type: non-prod
        version: qa
      - env: integration-non-gpu
        region: us
        type: non-prod
        version: qa
[...snip…]    
      - env: prod-us
        region: us
        type: prod
        version: prod        
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
          path: my-chart
          targetRevision: HEAD
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

In the example above, each deployment depends on different factors according to the application version, if it’s production or not, its region, and so on.

This application set actually creates ALL our applications in a single step.

Again, if you look at the resources of each individual application, you see each one has a different configuration according to all Helm values “merged” in the file tree.

Using a Helm repository with a hierarchy of Helm values

In all the previous examples, we used the Helm chart from Git, which is also our recommendation. However, we know some teams like to use OCI registries or Helm repositories. You can modify the previous Application Set and refer to a Helm repository instead of a Git folder for the Helm chart.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs-from-repo
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - env: integration-gpu
        region: us
        type: non-prod
        version: qa
[...snip…]
      - env: prod-us
        region: us
        type: prod
        version: prod        
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://kostis-codefresh.github.io/multi-sources-example
          chart: my-chart
          targetRevision: 0.1.0
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'


      # Sync policy
      syncPolicy:
        syncOptions:
          - CreateNamespace=true  
        automated:
          prune: true
          selfHeal: true

You can see that we now specifically define the 0.1.0 chart from a Helm repo hosted in GitHub pages instead of loading the chart from a Git revision.

Other than that, the application set is exactly the same. The end result after deployment is also the same as the previous example.

Saving environment configurations on their own files

The last example has a very long list generator. As you add more environments, the list will become even longer. It’s far easier to extract this useful information in individual files. We can use the Git file generator to achieve this.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: all-my-envs-from-repo-with-version
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - git:
      repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
      revision: HEAD
      files:
      - path: "appsets/4-final/env-config/**/config.json"      
  template:      
    metadata:
      name: '{{.env}}'
    spec:
      # The project the application belongs to.
      project: default


      sources:
        - repoURL: https://kostis-codefresh.github.io/multi-sources-example
          chart: my-chart
          targetRevision: '{{.chart}}'
          helm:
            valueFiles:
            - $values/my-values/common-values.yaml          
            - $values/my-values/app-version/{{.version}}-values.yaml              
            - $values/my-values/env-type/{{.type}}-values.yaml  
            - $values/my-values/regions/{{.region}}-values.yaml                
            - $values/my-values/envs/{{.env}}-values.yaml
        - repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
          targetRevision: HEAD
          ref: values
     
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.env}}'

In this example, we extracted all environment configuration in config.json files. We also templated the chart version, so we can define differently for each environment.

This lets you edit each environment individually. For example, if we want chart version 0.1.0 in all production environments and 0.2.0 in all other environments. 

Here is the configuration for prod-us:

{
    "env": "prod-us",
    "region": "us",
    "type": "prod",
    "version": "prod",
    "chart": "0.1.0"
  }

And the one for QA:

{
    "env": "qa",
    "region": "us",
    "type": "non-prod",
    "version": "qa",
    "chart": "0.2.0"
  }

Now you have a very flexible Argo CD setup that can cover most scenarios with minimal effort.

Scenario 1 – Bootstrap all apps in a single step -> Deploy the Application Set

Scenario 2 – Add a new setting to all European deployments -> change the file my-values/regions/eu-values.yaml

Scenario 3 – Making a common change to all environments apart from production -> change the file my-values/env-type/non-prod-values.yaml

Scenario 4 – Change chart version in Production US -> change the file env-config/prod/us/config.json

Scenario 5 – Make the QA environment inherit settings from Europe region instead of US -> change the file env-config/qa/config.json

Scenario 6 – Create a brand new environment called load testing with the same settings as Integration (GPU) -> copy the folder env-config/integration/gpu/ to env-config/load-testing and edit the config.json file accordingly. 

Conclusion: Start simple

In this guide, you’ve seen how to combine different configuration settings with your Helm charts and how to employ Helm value hierarchies to model your environments without duplication. You also learned the proper way to use the multi-source feature of Argo CD.

We’ve shown that plain Application Sets can go a long way toward modeling all usual cases, even with Helm charts. You don’t need to template Application Sets with an extra Helm layer, and you certainly don’t need to resort to hacks for “default” and “extra” values.

We recommend using this structure as a starting point in your organization. Only add complexity if it’s really required by your business needs.

4 thoughts on “Using Helm Hierarchies in Multi-Source Argo CD Applications for Promoting to Different GitOps Environments

  1. Thanks for this guide, it was very valuable in helping me think about my own strategy. I have 3 environments, I’m using a chart for each of my company’s applications and I don’t want to package them with helm, I want to use git. I love this example, but it’s not clear to me how to adapt it to multiple applications, should I have a repo with this structure for each application or can I include all my apps within this same repository?

    If I go for the second option, instead of using my-chart should I use my-chart-app1, my-chart-app2, my-chart-app3 and the same with the values directory ​​? values-app1, values-app2, values-app3 ? and then with argocd call the necessary chart according to the application and point to the corresponding values ​​directory using ‘sources’ ?

    What do you suggest as best practice?
    thanks!

    1. Hello. You can have multiple applications. In a single git repository. For that you need to modify the last ApplicationSet shown and add a matrix generator that combines two generators (files for apps, and environments)

      This has already been shown in the Application Set guide https://codefresh.io/blog/how-to-structure-your-argo-cd-repositories-using-application-sets/

      You should also explore having a single chart for your all apps to simpify things. But what you suggest (different chart per app) is also possible. Values must definitely be different per app.

  2. Great and very useful post Kostis,

    I have a question.
    Now, I am trying to implement it to deploy Nexus Artifactory in non-prod and prod in two Azure regions.
    Before reading the article, I created a repo with a Helm chart that only has the values-non-prod-region1.yml, values-prod-region1.yml files. In the chart.yml I only have the version and a dependency to the nexus-ha Helm chart. I did it in this way because I have to deploy extra YAMLs that are in repo/templates.

    Is there a way to add YAML files to the repository for deployment without structuring it as a Helm chart?
    In other words, how would you include the additional YAML files for deployment in your second source?

    Thanks in advance

    1. When you say “I created a repo with a Helm chart”, do you mean a Git repo or a Helm repo?

      I would store everything in Git.

      “In other words, how would you include the additional YAML files for deployment in your second source?”

      I would reference this from Git as shown in the article. You can include as many YAML files as you want like this.

      Or are you asking about something different?

Leave a Reply

Your email address will not be published. Required fields are marked *

Comment

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments