One of Argo CD’s standout features is its powerful user interface (UI) that shows the live status of all applications and the respective Kubernetes resources. Both developers and operators can quickly understand the status of their deployments by looking at the UI and drilling down into all the different views of Argo CD applications. Some teams even use the Argo CD UI as a generic Kubernetes dashboard and management interface (especially after the addition of the web-based terminal feature).
As organizations scale their Argo CD use, they understand that using a single Argo CD instance across separate teams is only possible with proper security controls. Argo CD is designed to handle multi-tenant installations and has comprehensive controls for limiting and isolating applications between developer teams. This lets operators define who has access to what application and with what limits.
In this guide, we examine the individual building blocks (application projects, security policies, user groups, and roles) and explain how you can set up a multi-tenant Argo CD installation for your organization.
Popular use cases for multi-tenant Argo CD installations
Each organization has different requirements when it comes to risk and ease of management. We’ve seen many ways that teams prefer to use (or not use) Argo CD in a multi-tenant/multi-use environment.
Firstly, we have the use case where developers have full access to the Argo CD UI and can debug/sync/edit their deployments with full power. This is great if your developers already have some Kubernetes knowledge and know the principles of GitOps.
After your team reaches a certain size, you need some restrictions in place, especially with larger developer teams. The classic scenario is where developers have full access to non-production environments, but read-only access to production environments. They can look at the status of production and possibly retrieve logs, but can’t change anything on the deployment.
The restrictions on developers can be as strict as you want. Argo CD supports isolating developers to specific namespaces and clusters, hiding completely unrelated applications from the UI, and even setting up different access for different application actions. For example, a developer might have access to take logs from an Argo CD application, but they’re restricted from triggering a re-sync of the application.
At the other end of the spectrum, some organizations choose to hide Argo CD completely from developers. Argo CD is reserved for operators only. Developers interact with their applications using an external system that often sits on top of Argo CD and abstracts several Kubernetes concepts (like an internal developer platform or portal).
In all scenarios, it’s important to decide whether you let developers to create new applications on their own or only edit existing ones. You also need to consider how to handle ephemeral environments.
Three important things to remember are:
- Argo CD comes with its own role-based access control (RBAC) mechanisms that aren’t related to Kubernetes RBAC.
- There are several moving parts that coordinate together (single sign-on (SSO), roles, app-projects).
- This guide is about your internal users. If you want to achieve multi-tenancy for external users and customers, you probably need to look at other approaches.
There isn’t a one-size-fits-all solution, of course. It’s also impossible to cover all variations in a single guide. There are several designs that you will see in real life.
Consider this guide a gentle introduction to the security controls of Argo CD. We explain the theory behind the security controls and share a semi-realistic example to give you ideas on how to implement multi-tenancy in your own organization.
Argo CD security overview
There are two different aspects when it comes to multi-tenant Argo CD installations:
- Limits over where applications get deployed. Which Kubernetes resources can people deploy and from which Git repos.
- Security over Argo CD resources. Who can see an application, sync it, delete it etc.
These two aspects are independent and are linked together via the application project resource.
The other pieces of the puzzle are:
- Users, groups, and SSO configuration (single sign-on)
- Local users (optional)
- Global security rules and per-project security policies
- Grouping of policies into roles
- Argo CD resource actions
- Other associated application resources such as clusters, namespace, and Git repos
We briefly cover all these components in the next sections.
How to define access for Argo CD resources
Let’s start with Argo CD resources (i.e., applications). The RBAC model behind Argo CD resources follows the who-what-how model. Resources get secured via RBAC policies. Essentially, you can define who has access to what application and how they can access it.
Even though Argo CD can define security controls with individual objects (“allow David to edit application billing”), in reality, it’s most useful to treat similar objects in groups. Therefore, all individual parts of a policy are mostly applied between groups instead of individual entities.
After you have all groups/roles/apps defined, you can apply policies on a global or project level.
You define policies in the casbin format and follow the syntax:
p, <role/user/group>, <resource>, <action>, <object>, <effect>
The first p stands for “policy”.
Here are some examples of policies:
“p, infra:read-only, applications, get, infra/*, allow”
This policy says, “All users that belong to the infra/read-only group are allowed to view/get all applications that belong to the infra app-project”.
Another example:
“p, developers:full-access, applications, delete, prod/*, deny”
This policy says, “All users that belong to the developers:full-access group are NOT allowed to delete applications that are part of the prod app-project”.
You can find all the details for what actions and resources are available in the official Argo CD documentation.
The important point here is that you need to do some analysis and split all your Argo CD applications into app projects. Different organizations will select different meanings for the projects. A project can be a specific team of people, a department, “environment”, or anything else that makes sense in your organization for application groups. We recommend using projects for each team (both developer teams and infra teams).
After you’ve defined all your policies, you should store them declaratively in your application projects under .spec.roles.
The second important point is that contrary to other security designs, a “role” in Argo CD is a set of policy resources. You first create an app project, then apply policies to it, and finally connect policies to users.
This means you need to anticipate how your applications will get used and design appropriate roles. For example, if you follow the scenario where a group of developers can see all applications in Argo CD but in “view-only” mode, then you need to create a “view-only” security policy in all the respective application projects.
How to define access for Kubernetes resources
RBAC policies define how users can interact with the main Argo CD resources (applications). The application project has additional capabilities for securing Kubernetes resources.
Remember that an Argo CD application is, in its essence, just a link between a Git repository and a Kubernetes cluster/namespace. In a multi-tenant environment, you almost always want to restrict:
- What clusters/namespaces developers can deploy to
- Which Git repos/paths can be used for applications
- What kind of Kubernetes resources can be affected
- Optionally, when deployments can happen
The default project is an “allow-everything” project that has no restrictions at all.
spec:
sourceRepos:
- '*'
destinations:
- namespace: '*'
server: '*'
clusterResourceWhitelist:
- group: '*'
kind: '*'
Using the default project is great for experimenting with Argo CD, but we recommend you don’t use it in a production setup and only have your own projects defined.
Application projects are very flexible when it comes to cluster resources. You can isolate developers to specific namespaces, specific types of resources, and days of deployments using sync windows.
Here is an example application project:
apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: team-revenue namespace: argocd spec: description: Team with developers for Revenue sourceRepos: - "https://github.com/acme/revenue/*" destinations: - namespace: "revenue" server: https://kubernetes.default.svc clusterResourceBlacklist: - group: "" kind: "Namespace" namespaceResourceBlacklist: - group: "" kind: "ResourceQuota" - group: "networking.k8s.io" kind: "NetworkPolicy"
This project defines what the “Revenue” team can do.
- They can only deploy from Git repos under acme/revenue.
- They can only use the namespace “revenue” in the local cluster.
- They cannot create new namespaces, resource quotas, and network policies.
While the argocd CLI includes several commands for managing projects, again we recommend you declare everything in the Project CRD and use a declarative setup for a production environment.
A complete example with two developer teams
We’ve seen how the application project is the centerpiece of Argo CD security in multi-tenant environments.
Let’s take a look at a full example. In our imaginary company, we have 2 developer teams (A and B) that continuously deploy applications. We also have infrastructure applications (i.e., cert-manager) handled by the operator team.
Here are our requirements:
- Admin/operator can view/edit all applications (including infra apps).
- Team B can only view/edit their own applications and nothing else.
- Team A can view/edit their own applications and view-only applications from team B.
- No developer from either team can view the infrastructure applications.
- No developer can delete any application (even their own).
- All applications (including infra) can only be synced from a specific GitHub organization/user.
You can find all Argo CD settings at https://github.com/kostis-codefresh/intro-argocd-rbac . Note that for simplicity, we used local users instead of SSO.
We have 4 applications in total – one for each team and 2 infrastructure apps. The admin of Argo CD can see all of them and can perform any action.
Developers from team B are the most constrained. They can only look at their application and nothing else.
They can sync at will, but they cannot delete the application. Pressing the delete button presents an error message.
Developers from team A can see both apps from both teams, but not the infrastructure apps.
Even though they can sync/edit their own application, they cannot affect the application of team B. If they try to perform a sync on the application of team B, again they will get an error message.
With this set up, we’ve satisfied our initial requirements. Let’s look under the hood to see how everything was implemented.
Using app-projects per team
We mentioned in the introduction using application projects to represent teams. In our example, we defined 3 projects, one for each team (A, B, and infra).
The most interesting project is the one from team B:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-b
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
description: Team B
destinations:
- name: '*'
namespace: '*'
server: '*'
roles:
- description: Read Only
name: read-only
policies:
- p, proj:team-b:read-only, applications, get, team-b/*, allow
- description: Read Write
name: read-write
policies:
- p, proj:team-b:read-write, applications, get, team-b/*, allow
- p, proj:team-b:read-write, applications, sync, team-b/*, allow
- p, proj:team-b:read-write, applications, update, team-b/*, allow
sourceRepos:
- https://github.com/kostis-codefresh/intro-argocd-rbac.git
Notice that we define 2 roles here (sets of policies). One is the “read-write’ for team B, and one is “read-only” for team A. Notice also that neither role includes a “delete” action, This means that no developer can delete the applications of this project. Finally, we restrict the application source repository to a specific GitHub project.
After we have AppProjects in place, we deploy all 4 applications using an application set. If you’re not familiar with application sets, please read our application sets guide first.
The applicationset will automatically assign each application to the same AppProject as its parent folder.
This concludes the initial application deployment.
Assigning users to groups/roles
For simplicity, we use local users in our example. In a real system, you’d use SSO to assign users to groups for Argo CD.
In our case, we add 2 users (one for team A and one for team B) in argocd-cm.
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
data:
accounts.jane: apiKey, login
accounts.jane.enabled: "true"
accounts.david: apiKey, login
accounts.david.enabled: "true"
You can set the password for local users with the argocd account update-password command or via declarative setup.
Finally, we assign each user to their appropriate roles in argocd-rbac-cm.
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
namespace: argocd
labels:
app.kubernetes.io/name: argocd-rbac-cm
app.kubernetes.io/part-of: argocd
data:
policy.csv: |
g, jane, proj:team-a:read-write
g, jane, proj:team-b:read-only
g, david, proj:team-b:read-write
You can see here that Jane belongs to team A. She has read/write access to all apps of team A and read-only access to projects from team B. David is on team B and he can only see his own projects.
Neither Jane nor David see any of the applications from the “infra” projects (which was one of the original requirements).
This concludes the setup. If you now log in as Admin, Jane, or David in the Argo CD user interface, you’ll see only what each user is allowed to view/edit.
Where to go from here
This guide was a gentle introduction to Argo CD RBAC and how to use Argo CD in multi-user situations. There are several more things to explore, like appProject hierarchies, specific SSO configurations, complex policies, and more. For now, you should have some great ideas for your own organization and understand how all the different components work together to achieve multi-tenancy.
Security is a complex topic. Consider the present guide a starting point for your organization. You should always follow the best practices and requirements of your security team.