Selectors for concurrency synchronization

Argo Workflows has a synchronization mechanism to limit parallel execution of specific workflows or templates within workflows, as required.
The mechanism enforces this with either semaphore or mutex synchronization configurations. For detailed information, see Synchronization.

Codefresh supports an additional level of concurrency synchronization, with selectors, for both workflows and templates.

Selectors enable access to all workflow properties such as annotations, labels, or parameters using Go template formats. Selector-based concurrency configurations support parallel executions of workflows or templates within workflows. And equally important, selectors work with both semaphore and mutex synchronization configurations.

This article has examples for semaphore synchronization configurations.

Semaphore synchronization is configured in the ConfigMap, which is referenced from workflows or templates within workflows.

Here’s an example of the ConfigMap with the concurrency definition:

apiVersion: v1
kind: ConfigMap
metadata:
 name: semaphore-config
data:
 workflow: "1"

Concurrency selector definitions

A concurrency synchronization selector is defined through a name-template pair in workflows or templates:

• The name is any meaningful/logical name that describes the selector. For example, Git repo or branch.
• The template is the parameter mapping for the name, that is resolved to the selector value when the pipeline is run. The parameter mapping format must confirm to Sprig templates. For information on Go templates and Sprig functions, see the Go template package documentation, and Sprig Function Documentation. For example:

{{ workflow.parameters.REPO_OWNER }}/{{ workflow.parameters.REPO_NAME }}

or

{{ workflow.parameters.GIT_BRANCH }}

Selectors are added to the synchronization section, under selectors, as in the sample YAML below.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
 generateName: synchronization-wf-2
spec:
 entrypoint: whalesay
 arguments:
   parameters:
   - name: REPO_OWNER
     value: denis-codefresh
   - name: REPO_NAME
     value: argo-workflows
   - name: GIT_BRANCH
     value: main
 synchronization:
   semaphore:
     configMapKeyRef:
        name: semaphore-config
        key: workflow
     selectors:
      - name: repository
        template: "{{ workflow.parameters.REPO_OWNER }}/{{ workflow.parameters.REPO_NAME }}"
      - name: branch
        template: "{{ workflow.parameters.GIT_BRANCH }}"
 templates:
   - name: whalesay
     container:
       image: docker/whalesay:latest
       command: [sleep]
       args: ["5000"]

When and how does selector concurrency configuration work?

Let’s review a viable use case for selector-based concurrency synchronization, and different scenarios to illustrate when and how it works:

• Default concurrency synchronization without selectors
• Selector concurrency synchronization resolving to the same template values
• Selector concurrency synchronization resolving to different template values

Use case

We have two workflows, synchronization-wf-1 and synchronization-wf-2.

Both workflows commit to the same Git repo, but to different branches in the repo. The first workflow commits to the main branch, and the second workflow commits to a feature branch. When there are commits to the main and feature branches, we want the workflows to run in parallel.

Default concurrency synchronization

Both workflows have the same default values for the arguments, and share the same semaphore configuration and key with weight 1.

With the default concurrency configuration, only one workflow at a time can use the key. Meaning that while synchronization-wf-1 is Running, synchronization-wf-2 is in Pending status.

Selector concurrency synchronization resolving to same template values

Here are the same workflows, synchronization-wf-1 and synchronization-wf-2, this time configured with selector concurrency.
Both workflows share the same semaphore key with a weight of 1.
Both workflows also use the same set of selectors. As you can see in the image below, the Git repo and the Git branch arguments have the same values in both.

In this case, the selectors have no impact as the selector templates in both workflows resolve to the same values, resulting in the same semaphore key. If you look at the status of both workflows, you can see that they have the same semaphore key.

synchronization:
 semaphore:
   holding:
   - holders:
     - synchronization-wf-6lf8b
     semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=main

Selector concurrency synchronization resolving to different template values

In the third scenario, we have the same workflows, synchronization-wf-1 and synchronization-wf-2, also configured with the selector concurrency.
They also share the same semaphore configuration and key with weight 1.

The difference is that in this scenario, the selector templates in both workflows resolve to different values, resulting in different semaphore keys for each workflow.
In synchronization-wf-1, the Git branch is set to main, and in synchronization-wf-2, it is set to feature.

Now, because the workflows use different semaphore keys, both workflows run in parallel.
If you look at the status of both workflows, you can see that they have different semaphore keys.

synchronization:
 semaphore:
   holding:
   - holders:
     - synchronization-wf-6lf8b
     semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=main

synchronization:
 semaphore:
   holding:
   - holders:
     - synchronization-wf-8lf9b
     semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=feature

Related articles

Creating workflows