ArgoCD Secrets: Two Technical Approaches, Step By Step

What Is Argo CD?

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. By using Git repositories as the source of truth for Kubernetes cluster configurations, Argo CD automates the deployment of applications. This automation ensures consistency in environments, minimizing human error and manual intervention. 

The use of a Git repository allows configurations to be version controlled, tracked, and audited over time. Argo CD offers features such as automated deployment, rollback capabilities, and integration with popular identity providers for secure operations. 

Arpo CD constantly monitors new Git commits, verifying real-time application statuses against the intended state. This monitoring extends to providing real-time alerts if discrepancies arise, allowing for immediate corrective actions.

What Are Kubernetes Secrets?

Kubernetes secrets are a mechanism for storing sensitive data, such as passwords, OAuth tokens, and SSH keys, in a Kubernetes cluster. They offer a secure way to manage information that should not be stored in code or non-secure storage areas. 

With secrets, Kubernetes allows developers to build more secure applications by granting access to sensitive data only to authorized pods and containers running within the cluster. This granular security control is important for maintaining data protection standards.

Secrets in Kubernetes are encoded, providing a minimal level of security over plain text but not true encryption. Although base64 encoding is used, it’s crucial to implement additional security measures like encryption and restricted access, ensuring that sensitive data stays protected from unauthorized access.

Read our guide to Argo Kubernetes

Options for Secret Management in Argo CD

Argo CD does not enforce a particular method for managing secrets. This allows teams to integrate a variety of secret management tools and strategies into their GitOps workflows, depending on their security needs and infrastructure.

Common approaches for secret management in Argo CD involve using plugins to inject secrets into application manifests at deployment time. However, it is important to ensure that these plugins are securely configured to avoid potential security risks. Some popular tools for managing secrets in a GitOps setup with Argo CD include:

1. Sealed Secrets: Kubernetes controller that allows secrets to be encrypted and stored safely in git repositories.
2. External secrets operator: Syncs secrets from external secret management systems like Hashicorp Vault into kubernetes.
3. SOPS (Secret Operations): An open source tool that encrypts secrets and make them available in a secure manner within a Kubernetes environment.
4. Helm secrets: Integrates secret management directly into Helm charts.
5. Kustomize secret generator plugins: Extends Kustomize to handle secrets in a templated manner.
6. Other options: It is also possible to manage secrets via the Kubernetes secrets store CSI driver and specialized Argo CD plugins like argocd-vault-plugin and argocd-secret-replacer. 

In the rest of this article we’ll cover the first two options for managing secrets with Argo CD.

Option #1: Using Sealed Secrets with Argo CD

Sealed Secrets provides a secure and Kubernetes-native method for managing secrets in GitOps workflows, including Argo CD environments. Sealed secrets ensure that sensitive data like API keys, passwords, and certificates remain protected, even when stored in Git repositories. This is achieved by encrypting the secrets before they are committed to Git and then decrypting them during deployment.

Workflow of Sealed Secrets with Argo CD:

1. Encrypt secrets: Using the kubeseal CLI tool, developers can encrypt a Kubernetes secret and convert it into a SealedSecret resource. The SealedSecret can safely be stored in the Git repository as it is encrypted using a public key managed by the Sealed Secrets controller.

For example:

echo -n bar | kubectl create secret generic examplesecret --dry-run=client --from-file=foo=/dev/stdin -o yaml  > examplesecret.yaml

We can check the contents of the created examplesecret.yaml by using the following command:

nano examplesecret.yaml

kubeseal --format yaml -f examplesecret.yaml > examplesealedsecret.yaml

You can view the content of examplesealtedsecret.yaml by running the following command:

nano examplesealtedsecret.yaml

kubectl create -f examplesealedsecret.yaml

Confirm the secret has been created using the following kubectl command:

kubectl get secrets

2. Deploy sealed secrets with Argo CD: In Argo CD, users can deploy applications that include SealedSecret resources by storing them in the same repository as the Helm charts or Kubernetes manifests. During the deployment process, the Sealed Secrets controller will automatically decrypt the SealedSecret into a Kubernetes secret, making it available to the application without exposing sensitive information in plain text.

3. Proxy chart pattern: A useful technique when using Sealed Secrets in Argo CD is to implement a proxy chart. This chart allows developers to reference and customize Helm charts while managing the SealedSecret resources independently. The SealedSecrets are stored in the templates directory of the proxy chart, which simplifies version control and enhances flexibility in how secrets are applied across environments.

Example structure:

├── Chart.yaml
├── templates
│   └── sealed-secret.yaml
└── values.yaml

Key benefits of Sealed Secrets with Argo CD:

• Kubernetes-native: Sealed Secrets integrate into Kubernetes, allowing users to manage secrets directly within Kubernetes workflows.
• Prevent value overriding: Encrypting secrets ensures that values remain consistent and secure, avoiding accidental overwriting of sensitive data.
• Multi-cluster secret sharing: It is possible to reuse the same encrypted secret across multiple clusters, simplifying secret management in complex infrastructures.
• Secure GitOps: Storing encrypted secrets in Git ensures that sensitive data remains protected, aligning with best practices for version control and security.

Option #2: Using Argo CD with an External Secret Manager

Argo CD can integrate with external secret management solutions to securely manage sensitive data without storing secrets directly in Git repositories. One popular method is to use the External Secret Operator (ESO), which acts as a Kubernetes controller to retrieve secrets from external providers like AWS, Azure, Google Cloud, or HashiCorp Vault. 

These secrets are then converted into Kubernetes-native secrets, which applications can consume without exposing sensitive information in GitOps workflows.

Key features of External Secret Operator:

• Avoiding secrets in Git: With ESO, sensitive information like API keys or database credentials can be stored in external systems (e.g., Vault), not in Git. This solves the issue of storing sensitive tokens in Git repositories, which could otherwise expose them to unauthorized access.
• Automatic secret refreshing: ESO can automatically update secrets without restarting Kubernetes pods or redeploying applications. This allows for seamless secret rotation, critical for maintaining security.

Workflow of External Secret Operator with Argo CD:

1. Define the secret source: The ESO controller requires a custom resource definition (CRD), such as SecretStore or ClusterSecretStore, that specifies the source of the secret and how to authenticate with it. For instance, if using HashiCorp Vault, you create a ClusterSecretStore defining the Vault server and authentication method (e.g., Kubernetes service accounts).
   
   Here is an example configuration for a Vault-based secret store:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "demo"

2. Fetch secrets: After configuring the secret store, the ExternalSecret CRD is used to specify which secrets to retrieve from the external provider. ESO fetches these secrets and creates standard Kubernetes secrets that the application can access.
   
   Here is an example ExternalSecret for fetching database credentials:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-db-credentials
spec:
  refreshInterval: "20s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: mysql-credentials
  data:
    - secretKey: db_url
      remoteRef:
        key: mysql_credentials
        property: url
    - secretKey: db_username
      remoteRef:
        key: mysql_credentials
        property: username
    - secretKey: db_password
      remoteRef:
        key: mysql_credentials
        property: password

3. Application deployment: During application deployment, Kubernetes secrets created by ESO are mounted in pods. The secrets can be accessed as files or environment variables, allowing the application to consume them securely. For example, the following configuration mounts the mysql-credentials secret in the application’s pod:

volumes:
- name: mysql
  secret:
    secretName: mysql-credentials

4. Secret rotation and refresh: ESO automatically refreshes secrets based on the defined refresh interval, ensuring that updated credentials or keys are reflected in running applications without requiring a pod restart or redeployment.

Codefresh: A Modern, Argo-Based CI/CD Platform

The Codefresh platform, powered by Argo, combines the best of the open-source with an enterprise-grade runtime allowing you to fully tap the power of Argo Workflows, Events, CD, and Rollouts. It provides teams with a unified GitOps experience to build, test, deploy, and scale their applications.

You can use Codefresh for your platform engineering initiative either as a developer portal or as the machinery that takes care of everything that happens in the developer portal. Your choice depends on how far your organization has adopted Kubernetes and micro-services

Codefresh is a next generation CI/CD platform designed for cloud-native applications, offering dynamic builds, progressive delivery, and much more.