Two of the most popular guides we’ve written are the GitOps promotion guide and the ApplicationSet guide. Used together, they explain an end-to-end solution for organizing your GitOps applications and promoting them between different environments, while keeping things DRY by using application sets.
Both of these guides use Kustomize. We offered some hints for Helm users, but this is a dedicated guide for Helm applications. Organizations that choose Helm have several other challenges to overcome when adopting GitOps. This is because you can also store Helm charts in dedicated Helm repos (apart from Git). Helm charts have versions on their own.
Unfortunately, we’ve seen teams try to replicate features in Argo CD that Helm already supports. Or, some teams try to adopt Argo CD without first understanding the basics of Helm.
In this guide, we include several Helm-related topics to consider when adopting GitOps and Argo CD in your organization.
We describe:
- The recommended Helm structure for GitOps repositories
- When to use the multi-source feature of Argo CD and when not to use it
- How to create Helm value hierarchies and why this is important
- Common Helm bad practices and misconceptions that people carry over to Argo CD
If you like our guides but wonder how our advice works with Helm and not Kustomize, then this guide is for you!
You can find the full example at https://github.com/kostis-codefresh/multi-sources-example.
How Helm values work
The original design for Helm is to have a generic chart (without any hardcoded configuration) and then use several different value files that hold environment-specific configuration. We’ve seen this scenario in detail in our guide about Helm values. It’s also possible to have default values for a chart if no external values are added.
You need to decide if your Helm chart will map to a specific application or not. One approach is to have a Helm chart for every application you develop. In this case, the Helm chart has 2 versions. One version keeps a history about the chart itself and the second version (appVersion in the Chart.yml file) keeps track of the application contained in the chart.
With the second approach, you have a reusable “single” chart that’s targeted at many applications. It doesn’t hold a specific application on its own. You always need to combine it with values that define the image/tag/configuration.
Both approaches are valid and Argo CD supports both. It makes sense, however, to select a single strategy for your organization.
Assuming the classic trinity of QA/Staging/Prod you should be able to easily install any chart with the appropriate configuration, like this:
helm install ./my-chart/ --generate-name -f values-qa.yaml helm install ./my-chart/ --generate-name -f values-staging.yaml helm install ./my-chart/ --generate-name -f values-production.yaml
In this very simple example, the charts are saved in Git as normal files along with their possible configurations.
Creating reusable configuration with Helm value hierarchies
The structure explained above works for most organizations. In some cases, however, the application has a complex configuration you need to define for a large number of environments. You could duplicate the values file for each environment, but this would create a cumbersome setup where you need to change common configuration between some environments.
To overcome this challenge, Helm also supports value hierarchies. This means that instead of defining a single value file when installing a chart, you can define multiple. Helm then merges all value files, and for properties defined multiple times, the last file wins.
This means that you can do:
helm install ./my-chart/ --generate-name -f common.yaml -f more.yaml -f some-more.yaml
If you have properties that are both in common.yaml and in more.yaml, the latter will prevail. In the same way, all properties defined in some-more.yaml will function as “overrides” for everything else you defined in the previous files.
The obvious advantage of this pattern is that you can create a hierarchy of Helm value files where “early” files contain common values for most environments and “late” files contain specific values for individual deployments.
A possible setup would be:
common-values.yaml
+-----all-prod-envs.yaml
+----specific-prod-cluster.yaml
This means common-values get shared between ALL your deployments unless they’re overridden in “all-prod-envs” for all your production environments, which can also be overridden by “specific-prod-cluster.yaml” for a specific cluster.
To make this work, you need to apply all files in the expected order – remembering the last one wins.
helm install ./my-chart/ --generate-name -f common-values.yaml -f all-prod-envs.yaml -f specific-prod-cluster.yaml
If you’re using Helm for your own applications, it’s essential you create an appropriate hierarchy for all your Helm applications that balances flexibility with minimal duplication.
It’s also important to remember that Helm has these hierarchies built-in. They’re unrelated to Argo CD.
Using Helm value hierarchies with Argo CD
After spending time designing your hierarchy of Helm values, you’re now ready to use Argo CD. The good news is that Argo CD supports Helm hierarchies out-of-the-box without any additional configuration.
Here’s an example application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
chart: my-chart
repoURL: https://github.com/my-example-app
targetRevision: 2.43
helm:
valueFiles:
- common-values.yaml
- all-prod-envs.yaml
- specific-prod-cluster.yaml
destination:
server: "https://kubernetes.default.svc"
namespace: default
The valueFiles property accepts an array of value files. Like the Helm CLI, each file mentioned in the list overrides all values from the previous files. The last file wins over the previous ones.
Why is this important? We’ve seen several teams try to replicate this pattern of default/environment-specific values using several hacks and workarounds, as we described in the previous guide. Usually, the teams that use these workarounds aren’t aware how Helm value hierarchies work.
Don’t fall into the trap of adding another layer of templating to your applications or appsets just to replicate Helm value hierarchies.
Using local Helm values with charts from Helm repositories
All the previous examples assume your Helm chart and its values are in Git. That’s always our recommendation, especially for your own applications. Some organizations, however, prefer to store charts in Helm repositories or OCI registries. Argo CD supports this scenario, but you need to understand the implications:
- Helm repositories (and OCI artifacts) are mutable. This means that you might download Helm chart 1.2.3 today with different contents than what you downloaded yesterday as version 1.2.3. And, of course, this breaks the GitOps principles.
- There’s no built-in auditing or history for changes made in the chart. If you use Git you get these features for “free”.
- There are risks regarding integrity and security. Can you verify that the chart you downloaded from the registry is the one the creator uploaded? Again, if you use your own Git repo, you can monitor and optionally approve all changes that happen on the chart.
If you insist on loading charts from external repositories, you should still keep your own values in your own Git repository. Argo CD can then merge the two using multiple sources.
Multiple sources is a feature of Argo CD created specifically for this scenario. In the most basic case, you should define 2 sources. One contains your external Helm chart and the other your Helm values.
Like before, you can also have Helm hierarchies.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-fancy-app
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: prod-us
sources:
- repoURL: https://acme.github.io/my-charts/
chart: my-app
targetRevision: 2.43
helm:
valueFiles:
- $values/common-values.yaml
- $values/all-prod-envs.yaml
- $values/specific-prod-cluster.yaml
- repoURL: 'https://github.com/example-org/all-my-values.git'
targetRevision: HEAD
ref: values
Notice here the relationship between the “ref:values” field and the “$values” prefix in the helm property. Of course, you can choose any name for the reference and not just “values”.
It should be clear that with this syntax, Argo CD has built-in support for Helm value hierarchies. Argo CD will load the chart from the external source and combine it with all the value files defined in order (last one wins).
On a related note, don’t abuse the multiple source feature of Argo CD. It’s NOT a generic way to group applications. Usually, you should have just 2 sources (as shown below) of related data that form a single application. If you find yourself grouping unrelated apps or having more than 2 to 3 sources, you need to look at app-of-apps and Application Sets instead.
Using Application Sets with Helm value hierarchies
If you deploy your Helm charts as an individual Argo CD application, you should do the same for your Application Sets. We covered Application Sets in detail in our previous guide.
Again, the key point here is that Application Sets also support Helm value hierarchies. And, like everything else in the application, you can template your chart values, folder, revision, or anything else you want.
Application Sets also work with multiple sources. So we can combine everything that we have seen so far in a single Application Set, like this:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: my-fancy-apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: qa
type: non-prod
- env: prod-eu
type: prod
- env: prod-us
type: prod
template:
metadata:
name: '{{.env}}'
spec:
# The project the application belongs to.
project: default
sources:
- repoURL: https://acme.github.io/my-charts/
chart: my-chart
targetRevision: 2.43
helm:
valueFiles:
- $values/common-values.yaml
- $values/all-{{.type}}-envs.yaml
- $values/{{.env}}-values.yaml
- repoURL: 'https://github.com/example-org/all-my-values.git'
targetRevision: HEAD
ref: values
# Destination cluster and namespace to deploy the application
destination:
server: https://kubernetes.default.svc
namespace: '{{.env}}'
This Application Set defines 3 environments (qa, prod-eu, prod-us). It uses Helm hierarchies and an external chart.
Notice that we can template the location of all value files without any additional templating solution.
We see many teams who place their application CRDs (or applicationSet CRDs) inside Helm charts to template them on a second level. This complexity isn’t needed. Application Sets on their own cover most common scenarios. Application Sets are a way of templating (the same as Helm). Using Helm to generate Application Sets (or Applications) is a recipe for disaster.
If there’s something you can’t template with Application Sets, it’s better to open an enhancement request instead of bringing yet another layer of templating into the mix.
A complete example of GitOps environments with Helm values
Let’s see how everything works with a semi-realistic example. You can find all the Kubernetes manifests and Argo CD resources at https://github.com/kostis-codefresh/multi-sources-example.
The repository contains:
- A generic Helm chart both in Git and as part of a Helm repo served by GitHub pages at https://kostis-codefresh.github.io/multi-sources-example/
- Helm values for different environments of the chart
- Example applicationsets that reference the Helm chart
- Example apps that use Helm hierarchies
The environments are similar to those defined in the promotion guide, but this time we use Helm values instead of Kustomize overlays.
We have the following environments:
- Integration testing (non-GPU and GPU-enabled)
- QA (only one instance, located in the US)
- Staging in EU, Asia, US
- Production only in EU and US
Similar to Kustomize reusable components, we also have a structure of reusable Helm values that you can use in a hierarchical manner.
The expectation is that to form a full Helm release, you need to combine multiple value files in order.
For example, to deploy the application to the “Staging” environment in the EU region, you need to do the following with plain Helm:
helm install ./my-chart/ --generate-name --create-namespace -n staging-eu \ -f my-values/common-values.yaml \ -f my-values/app-version/staging-values.yaml \ -f my-values/env-type/non-prod-values.yaml \ -f my-values/regions/eu-values.yaml \ -f my-values/envs/staging-eu-values.yaml
Remember that the last setting wins. You can always override the common settings with different configurations with files deeper in the tree.
Of course, we don’t want to use Helm, but install the same hierarchy with Argo CD. Here’s the respective Argo CD application.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: staging-eu-app
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: staging-eu
sources:
- repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
path: my-chart
targetRevision: HEAD
helm:
valueFiles:
- $values/my-values/common-values.yaml
- $values/my-values/app-version/staging-values.yaml
- $values/my-values/env-type/non-prod-values.yaml
- $values/my-values/regions/eu-values.yaml
- $values/my-values/envs/staging-eu-values.yaml
- repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
targetRevision: HEAD
ref: values
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
Notice again the hierarchy of value files. The file also follows the best practices outlined in our Application Set guide.
If you deploy this application with Argo CD, you can easily verify that the settings are correctly defined according to all the values “merged” in the correct order (the last one wins in case of duplicate properties).
That is great. But as we have said many times before, you shouldn’t deal with individual applications. We can also use Helm hierarchies with Application Sets.
There are many ways to slice and dice your apps. The following examples just show the possible combination of Helm hierarchies.
Application Set on a single dimension
Here’s a very simple Application Set that includes a list generator to model the regions for a single application.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: my-staging-appset
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- region: us
- region: eu
- region: asia
template:
metadata:
name: 'staging-{{.region}}'
spec:
# The project the application belongs to.
project: default
sources:
- repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
path: my-chart
targetRevision: HEAD
helm:
valueFiles:
- $values/my-values/common-values.yaml
- $values/my-values/app-version/staging-values.yaml
- $values/my-values/env-type/non-prod-values.yaml
- $values/my-values/regions/{{.region}}-values.yaml
- $values/my-values/envs/staging-{{.region}}-values.yaml
- repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
targetRevision: HEAD
ref: values
# Destination cluster and namespace to deploy the application
destination:
server: https://kubernetes.default.svc
namespace: 'staging-{{.region}}'
# Sync policy
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
Here, the Helm value locations are templated according to the application’s region. An alternative is to use the list generator for a list of apps, clusters, or environment types.Deploying this application set creates 3 applications, one for each region.
Again, you can easily verify that each deployment gets the appropriate settings according to the respective Helm value hierarchy.
Application Set with multiple templated values
In the previous example, we only looped on the “region” property of the value folders. We can use the list generator to loop over several different properties.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: all-my-envs
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: integration-gpu
region: us
type: non-prod
version: qa
- env: integration-non-gpu
region: us
type: non-prod
version: qa
[...snip…]
- env: prod-us
region: us
type: prod
version: prod
template:
metadata:
name: '{{.env}}'
spec:
# The project the application belongs to.
project: default
sources:
- repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
path: my-chart
targetRevision: HEAD
helm:
valueFiles:
- $values/my-values/common-values.yaml
- $values/my-values/app-version/{{.version}}-values.yaml
- $values/my-values/env-type/{{.type}}-values.yaml
- $values/my-values/regions/{{.region}}-values.yaml
- $values/my-values/envs/{{.env}}-values.yaml
- repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
targetRevision: HEAD
ref: values
# Destination cluster and namespace to deploy the application
destination:
server: https://kubernetes.default.svc
namespace: '{{.env}}'
In the example above, each deployment depends on different factors according to the application version, if it’s production or not, its region, and so on.
This application set actually creates ALL our applications in a single step.
Again, if you look at the resources of each individual application, you see each one has a different configuration according to all Helm values “merged” in the file tree.
Using a Helm repository with a hierarchy of Helm values
In all the previous examples, we used the Helm chart from Git, which is also our recommendation. However, we know some teams like to use OCI registries or Helm repositories. You can modify the previous Application Set and refer to a Helm repository instead of a Git folder for the Helm chart.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: all-my-envs-from-repo
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- env: integration-gpu
region: us
type: non-prod
version: qa
[...snip…]
- env: prod-us
region: us
type: prod
version: prod
template:
metadata:
name: '{{.env}}'
spec:
# The project the application belongs to.
project: default
sources:
- repoURL: https://kostis-codefresh.github.io/multi-sources-example
chart: my-chart
targetRevision: 0.1.0
helm:
valueFiles:
- $values/my-values/common-values.yaml
- $values/my-values/app-version/{{.version}}-values.yaml
- $values/my-values/env-type/{{.type}}-values.yaml
- $values/my-values/regions/{{.region}}-values.yaml
- $values/my-values/envs/{{.env}}-values.yaml
- repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
targetRevision: HEAD
ref: values
# Destination cluster and namespace to deploy the application
destination:
server: https://kubernetes.default.svc
namespace: '{{.env}}'
# Sync policy
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
You can see that we now specifically define the 0.1.0 chart from a Helm repo hosted in GitHub pages instead of loading the chart from a Git revision.
Other than that, the application set is exactly the same. The end result after deployment is also the same as the previous example.
Saving environment configurations on their own files
The last example has a very long list generator. As you add more environments, the list will become even longer. It’s far easier to extract this useful information in individual files. We can use the Git file generator to achieve this.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: all-my-envs-from-repo-with-version
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
revision: HEAD
files:
- path: "appsets/4-final/env-config/**/config.json"
template:
metadata:
name: '{{.env}}'
spec:
# The project the application belongs to.
project: default
sources:
- repoURL: https://kostis-codefresh.github.io/multi-sources-example
chart: my-chart
targetRevision: '{{.chart}}'
helm:
valueFiles:
- $values/my-values/common-values.yaml
- $values/my-values/app-version/{{.version}}-values.yaml
- $values/my-values/env-type/{{.type}}-values.yaml
- $values/my-values/regions/{{.region}}-values.yaml
- $values/my-values/envs/{{.env}}-values.yaml
- repoURL: 'https://github.com/kostis-codefresh/multi-sources-example.git'
targetRevision: HEAD
ref: values
# Destination cluster and namespace to deploy the application
destination:
server: https://kubernetes.default.svc
namespace: '{{.env}}'
In this example, we extracted all environment configuration in config.json files. We also templated the chart version, so we can define differently for each environment.
This lets you edit each environment individually. For example, if we want chart version 0.1.0 in all production environments and 0.2.0 in all other environments.
Here is the configuration for prod-us:
{
"env": "prod-us",
"region": "us",
"type": "prod",
"version": "prod",
"chart": "0.1.0"
}
And the one for QA:
{
"env": "qa",
"region": "us",
"type": "non-prod",
"version": "qa",
"chart": "0.2.0"
}
Now you have a very flexible Argo CD setup that can cover most scenarios with minimal effort.
Scenario 1 – Bootstrap all apps in a single step -> Deploy the Application Set
Scenario 2 – Add a new setting to all European deployments -> change the file my-values/regions/eu-values.yaml
Scenario 3 – Making a common change to all environments apart from production -> change the file my-values/env-type/non-prod-values.yaml
Scenario 4 – Change chart version in Production US -> change the file env-config/prod/us/config.json
Scenario 5 – Make the QA environment inherit settings from Europe region instead of US -> change the file env-config/qa/config.json
Scenario 6 – Create a brand new environment called load testing with the same settings as Integration (GPU) -> copy the folder env-config/integration/gpu/ to env-config/load-testing and edit the config.json file accordingly.
Conclusion: Start simple
In this guide, you’ve seen how to combine different configuration settings with your Helm charts and how to employ Helm value hierarchies to model your environments without duplication. You also learned the proper way to use the multi-source feature of Argo CD.
We’ve shown that plain Application Sets can go a long way toward modeling all usual cases, even with Helm charts. You don’t need to template Application Sets with an extra Helm layer, and you certainly don’t need to resort to hacks for “default” and “extra” values.
We recommend using this structure as a starting point in your organization. Only add complexity if it’s really required by your business needs.