Argo CD Notifications: From Basic to Advanced

What Are Argo CD Notifications? 

Argo CD Notifications is an extension of Argo CD that allows users to receive notifications regarding the status and activities of Kubernetes applications managed by Argo CD. The tool alerts users about specific events, such as application deployment successes, failures, and other significant changes in application status.

Notifications can be delivered via various channels, including email, Slack, and webhook. This ensures that users receive critical information through their preferred communication platforms. The system is configurable, enabling control over what types of events trigger notifications and how those notifications are formatted and sent.

How Do Argo CD Notifications Work? 

Argo CD Notifications integrate with the Argo CD application controller, which manages and monitors the state of applications:

1. The process begins with the Argo CD application controller, which continuously watches the repository for any changes or updates in the managed applications. 
2. When a change is detected, such as a code update or a deployment event, the application controller triggers an event. 
3. This event is then captured by the Argo CD Notifications controller. The notifications controller functions as a dedicated component that listens for these events. 
4. Once an event is captured, the controller processes it and determines whether it matches any predefined criteria for notifications. If it does, the controller forwards the event to the appropriate notification services.
5. These notification services are configured to send messages to various platforms, such as Slack, email, or custom webhooks. For example, when using Slack as the notification platform, the event data is formatted and sent to a specified Slack channel via the Slack notifications service.

Getting Started with Argo CD Notifications 

To start using Argo CD Notifications, you’ll need to install and configure the necessary components within your Argo CD environment. Here’s a step-by-step guide to get you up and running:

Step 1: Installing Triggers and Templates

Begin by installing the default triggers and templates provided by the Argo CD Notifications catalog. These predefined triggers and templates cover common notification scenarios and can be customized later as needed. Run the following command to install them:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications-catalog/install.yaml

This command applies a YAML configuration that sets up the necessary triggers and templates in the argocd namespace.

Step 2: Configuring Email Notification Service

To send notifications via email, you need to configure an email service by adding your email credentials to a Kubernetes secret. Here’s how to do it:

1. Add your email username and password to the argocd-notifications-secret:

EMAIL_USER=<your-email-user>
PASSWORD=<your-email-password>

kubectl apply -n argocd -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  email-username: $EMAIL_USER
  email-password: $PASSWORD
type: Opaque
EOF

2. Next, patch the argocd-notifications-cm ConfigMap to register the email service:

kubectl patch cm argocd-notifications-cm -n argocd --type merge -p '{
  "data": {
    "service.email.gmail": "{ 
      username: $email-username, 
      password: $email-password, 
      host: smtp.gmail.com, 
      port: 465, 
      from: $email-username 
    }"
  }
}'

This command adds the necessary configuration for sending emails using Gmail’s SMTP server.

Step 3: Subscribe to Notifications

Once the notification services are configured, you can subscribe to events for individual applications or projects. For example, to receive notifications in a Slack channel when a synchronization (sync) event succeeds, add the following annotation to your application:

kubectl patch app <example-app> -n argocd --type merge -p '{
  "metadata": {
    "annotations": {
      "notifications.argoproj.io/subscribe.on-sync-succeeded.slack": "<example-channel>"
    }
  }
}'

Replace <example-app> with the name of your application and <example-channel> with the relevant Slack channel.

Step 4: Test the Configuration

To verify that notifications are working, try syncing an application within Argo CD. If everything is set up correctly, you should receive a notification in your subscribed channel or email when the sync completes.

In some scenarios, you might want to enable notifications for applications across different namespaces. To do this, you’ll need to adjust the Argo CD Notifications controller configuration:

1. Update the controller’s startup parameters to specify which namespaces should be monitored and to enable self-service notifications:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
data:
  application.namespaces: app-team-one, app-team-two
  notificationscontroller.selfservice.enabled: "true"

2. You can deploy a ConfigMap named argocd-notifications-cm and an optional secret in the namespace where your Argo CD application resides. This setup allows users to configure their own notifications within their namespaces. For example:

apiVersion: v1
kind: Secret
metadata:
  name: argo-cd-notification-secret
type: Opaque
data:
  pagerduty-key-example-service: <base64-encoded-pagerduty-key>

Argo CD Notifications In-Depth 

Let’s dive deeper into some of the main concepts you should understand when working with Argo CD Notifications.

Triggers

Triggers in Argo CD Notifications define the conditions under which notifications should be sent. These conditions are specified using a trigger definition, which includes a name, a condition, and references to notification templates. The condition is a predicate expression that determines whether the notification should be sent. 

This evaluation is powered by the antonmedv/expr expression language, allowing for flexible and complex condition definitions. The triggers are configured in the argocd-notifications-cm ConfigMap. Here’s an example of a trigger that sends a notification when the application’s sync status changes to Unknown:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'  # Trigger condition
      send: 
        - app-sync-status
        - github-commit-status  # Template names

In this example, the trigger condition checks if the application’s sync status is Unknown. If this condition is met, the notification system uses the app-sync-status and github-commit-status templates to send notifications via the configured channels.

Condition Bundles

Triggers often bundle multiple conditions to handle various scenarios, each associated with different templates. This approach allows administrators to define triggers that cover multiple stages of an operation. For example:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  trigger.sync-operation-change: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: 
        - github-commit-status
    - when: app.status.operationState.phase in ['Running']
      send: 
        - github-commit-status
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: 
        - app-sync-failed
        - github-commit-status

This trigger handles different phases of a sync operation. Depending on the phase—Succeeded, Running, or Failed—different templates are used to generate notifications.

Avoiding Notification Flapping

In some cases, a trigger condition might cause “flapping,” where the status changes frequently, leading to repeated notifications. To prevent this, the oncePer field can be used to ensure notifications are sent only once per specific event or field change. For example:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  trigger.on-deployed: |
    when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
    oncePer: app.status.sync.revision
    send: 
      - app-sync-succeeded

This trigger sends a notification only once per observed Git revision, preventing repeated notifications if the application’s status fluctuates between Healthy and Progressing.

Default Triggers

Argo CD Notifications also supports default triggers, which are used if no specific trigger is defined in the subscription. Default triggers simplify the configuration for end users by providing a set of pre-defined triggers. For example:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  defaultTriggers: |
    - on-sync-status-unknown

  defaultTriggers.mattermost: |
    - on-sync-running
    - on-sync-succeeded

In this example, the default triggers are applied to channels like Slack or Mattermost, ensuring that notifications are sent for common events such as when a sync is running or has succeeded.

Templates

Templates define the content of the notifications that are sent out. These templates are configurable through the argocd-notifications-cm ConfigMap and are created using the Go html/template package, which allows for detailed customization of the notification messages.

Templates are reusable and can be referenced by multiple triggers.

Creating a Basic Template

A template typically includes a message format that can incorporate dynamic data related to the application. For example, the following template notifies users about the sync status of an application:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  template.example-custom-template-slack-template: |
    message: |
      Application {{ .app.name }} sync is {{ .app.status.sync.status }}.
      Application details: {{ .context.argocdUrl }}/applications/{{ .app.name }}.

In this template:

• {{ .app.name }} dynamically inserts the name of the application.
• {{ .app.status.sync.status }} provides the current sync status of the application.
• {{ .context.argocdUrl }}/applications/{{ .app.name }} gives the URL where more details about the application can be viewed.

Template Fields

Templates have access to several predefined fields that allow them to adapt to different scenarios:

• app: Holds the application object, giving access to all its metadata and status fields.
• context: A user-defined string map that can include any additional information required, such as URLs or environment names.
• secrets: Provides access to sensitive data stored in the argocd-notifications-secret, allowing secure handling of tokens or other credentials.
• serviceType: Specifies the notification service type (e.g., Slack, email) and can be used to render service-specific fields conditionally.
• recipient: Contains the name of the notification recipient.

User-Defined Context

The context field can be customized to share common information across multiple templates. This is useful when the same data, like environment details or region names, needs to be included in various notifications:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  context: |
    region: east
    environmentName: staging

  template.a-slack-template-with-context: |
    message: "Something happened in {{ .context.region }} in the {{ .context.environmentName }} data center!"

In this example, the context provides the region and environmentName, which are then dynamically inserted into the message.

Incorporating Secrets in Templates

Sometimes, it’s necessary to include sensitive information in notifications, such as webhook tokens. Argo CD Notifications handles this securely by allowing templates to reference secrets stored in the argocd-notifications-secret:

apiVersion: v1
kind: Secret
metadata: 
  name: argocd-notifications-secret
stringData: 
  sampleWebhookToken: secret-token
type: Opaque

This secret can be utilized in a template as follows:

apiVersion: v1
kind: ConfigMap
metadata: 
  name: argocd-notifications-cm
data: 
  template.trigger-webhook: |
    webhook:
      sample-webhook:
        method: POST
        path: 'webhook/endpoint/with/auth'
        body: 'token={{ .secret.sampleWebhookToken }}&variables[APP_SOURCE_PATH]={{ .app.spec.source.path }}'

In this setup:

• {{ .secret.sampleWebhookToken }} pulls the token from the secret, ensuring it’s used securely in the webhook request.
• {{ .app.spec.source.path }} dynamically includes the application’s source path in the webhook body.

Subscriptions

Subscriptions in Argo CD Notifications allow users to specify which recipients should receive notifications when certain events occur within Argo CD-managed applications. Subscriptions can be configured at different levels, such as individual applications, entire projects, or globally for all applications.

Defining Subscriptions for Individual Applications

Users can define subscriptions directly within an application’s annotations. This method is useful to subscribe specific recipients, such as Slack channels or email addresses, to notifications for particular triggers. Here’s an example of how to subscribe two Slack channels to notifications for a successful synchronization (on-sync-succeeded) event:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: 
  annotations: 
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: example-channel1;example-channel2

In this example:

• on-sync-succeeded is the trigger name that activates the notification.
• slack specifies the notification service to be used.
• example-channel1;example-channel2 is a semicolon-separated list of recipients, in this case, Slack channels.

Project-Level Subscriptions

Subscriptions can also be configured at the project level by annotating the AppProject resource.

This approach is useful for applying the same notification settings to all applications within a project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: 
  annotations: 
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: example-channel1;example-channel2

With this configuration, all applications under the AppProject will notify example-channel1 and example-channel2 of a successful sync event.

Global Subscriptions

For a more centralized approach, users can define global subscriptions in the argocd-notifications-cm ConfigMap. Global subscriptions apply to all applications, unless overridden by more specific annotations. Here’s how to configure global subscriptions:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  subscriptions: |
    # Notify slack:test2 and email:test@gmail.com for on-sync-status-unknown
    - recipients:
        - slack:test2
        - email:test@gmail.com
      triggers:
        - on-sync-status-unknown

    # Notify slack:test3 for on-sync-status-unknown, but only for apps with label test=true
    - recipients:
        - slack:test3
      selector: test=true
      triggers:
        - on-sync-status-unknown

In this example:

• recipients define the recipients, such as Slack channels or email addresses.
• triggers specify which events should trigger the notifications.
• selector allows users to filter which applications the subscription applies to, based on labels.

Using Webhooks in Subscriptions

If there is a need to send notifications via a custom webhook, it’s possible to store the webhook name in the recipients field of a subscription:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.webhook.<webhook-name>: |
    (snip)

  subscriptions: |
    - recipients:
        - <webhook-name>
      triggers:
        - on-sync-status-unknown

This configuration sends notifications to the specified webhook whenever the on-sync-status-unknown event occurs.

Functions

Argo CD Notifications provides a set of built-in functions that allow users to manipulate data within their notifications. These functions enable dynamic content generation, such as formatting strings, working with time values, and retrieving application details from repositories. They are particularly useful when creating custom notification templates, as they help adjust the message content based on the context of the application or event.

Time Functions

Time-related functions allow users to work with time data in notifications, making it possible to include timestamps or format dates appropriately:

• time.Now(): Returns the current time. This can be used to include the exact time when a notification is generated.
• time.Parse(val string): Parses a time string formatted according to the RFC3339 standard and returns a Time instance. This is useful when handling and displaying specific time formats.

String Functions

String functions help in formatting and manipulating string data within notifications. These functions are handy for adjusting case, replacing substrings, or making other textual modifications:

• strings.ReplaceAll(): Replaces all occurrences of a substring within a string with another value.
• strings.ToUpper(): Converts all characters in a string to uppercase, which can be used for emphasizing parts of the notification.
• strings.ToLower(): Converts all characters in a string to lowercase.

Sync Functions

Sync functions offer a way to retrieve detailed information about the application’s synchronization status. This is useful for customizing notifications based on the app’s state:

• sync.GetInfoItem(app map, name string): Retrieves information from the sync operation by using the name of the item.

Repository Functions

These functions extract and manipulate information related to the application’s source repository. They are helpful for creating notifications that include repository URLs, commit metadata, or other version control details:

• repo.RepoURLToHTTPS(url string): Converts a given Git URL into HTTPS format, making it more suitable for direct access in notifications.
• repo.FullNameByRepoURL(url string): Returns the full name of a repository in the format <owner>/<repoName>. This function currently supports repositories from GitHub, GitLab, and Bitbucket.
• repo.QueryEscape(s string): Escapes characters in a string to ensure it can be safely included in a URL. This is essential for constructing URLs that are part of notifications.
• repo.GetCommitMetadata(sha string): Fetches metadata for a specific commit from the repository. This metadata includes details such as the commit message, author, creation date, and associated tags.
• repo.GetAppDetails(): Provides details about the application’s source, including Helm, Kustomize, and directory specifications. This function returns fields like parameter names, value files, and other deployment-specific details.

Codefresh: A Modern, Argo-Based CI/CD Platform

The Codefresh platform, powered by Argo, combines the best of the open-source with an enterprise-grade runtime allowing you to fully tap the power of Argo Workflows, Events, CD, and Rollouts. It provides teams with a unified GitOps experience to build, test, deploy, and scale their applications.

You can use Codefresh for your platform engineering initiative either as a developer portal or as the machinery that takes care of everything that happens in the developer portal. Your choice depends on how far your organization has adopted Kubernetes and micro-services

Codefresh is a next generation CI/CD platform designed for cloud-native applications, offering dynamic builds, progressive delivery, and much more.

Ready to Get Started?

Deploy more and fail less with Codefresh and Argo

Sign Up