Argo Workflows: The Basics and a Quick Tutorial

What Is Argo Workflows?

Argo Workflows is an open source project that enables CI/CD pipeline management. It is a workflow engine that enables the orchestration of parallel jobs on Kubernetes.

Argo Workflows is implemented as a Kubernetes custom resource definition (CRD), which allows you to:

Define Kubernetes workflows using separate containers for each step in the workflow.
Model workflows with directed acyclic graphs (DAG) to capture dependencies between multiple steps or define task sequences.
Run compute-intensive data processing or machine learning tasks quickly and easily.
Natively run CI/CD pipelines on Kubernetes without configuring a complex application development solution.

This is part of an extensive series of guides about Kubernetes.

CI/CD with Argo Workflows on Kubernetes

Argo Workflows lets you define a YAML configuration with multiple steps, representing the steps in your CI/CD pipeline. Each of these steps runs in a separate container within your Kubernetes cluster.

Argo uses a CRD called Workflows, which provides a generateName. This name becomes the prefix of all the pods the Workflow runs. As part of the Workflow, you can also define storage volumes, which will be accessible by the templates for your workflow steps.

Once you set up a basic workflow configuration (learn more in the quick tutorial below), you can start defining the templates for each stage of the CI/CD process. A best practice is to define a central template with multiple steps, known as the entrypoint, where each step runs one of the templates in your workflow. Templates have input arguments, which you can use to pass data between the steps. You can have any number of steps, allowing you to automate complex CI/CD processes.

A major advantage of Argo Workflows is that it runs all pipeline steps on existing resources in the Kubernetes cluster, with no external dependencies. There is no need to spin up additional compute instances for the build server or other components.

Argo Workflows Core Concepts

This diagram illustrates the core concepts in Argo Workflows. Below we discuss each of these concepts in detail.

Image Source: Argo

Workflows

A Workflow is a central resource in Argo, defining the workflow to be executed and storing the workflow’s state. These critical functions mean you should treat Workflows as live objects—they are not merely static definitions but also serve as instances of a definition.

You define the workflow you want to execute in the workflow.spec template. The spec contains a list specifying the entry point and any templates. You can think of a template as a function that defines the instructions you want to execute. An entrypoint specifies the primary function of the workflow or the first template to execute.

TIPS FROM THE EXPERT

Use reusable templates to standardize tasks: Define reusable templates for common tasks such as code compilation, testing, and deployment. This ensures consistency across different workflows and simplifies maintenance.
Enable artifact management: Leverage Argo’s artifact repository capabilities to manage input and output artifacts. Store intermediate results and logs in a centralized location to facilitate debugging and auditing.
Optimize workflow efficiency with parallelism: Utilize Argo’s Directed Acyclic Graph (DAG) feature to parallelize independent tasks. This significantly reduces the total execution time compared to running tasks sequentially.
Secure your workflows with RBAC: Implement Role-Based Access Control (RBAC) to restrict who can create, modify, or delete workflows. This enhances security and ensures that only authorized personnel can make changes.
Monitor workflow execution with comprehensive logging: Enable detailed logging for all workflow steps. Using centralized logging solutions like ELK (Elasticsearch, Logstash, Kibana) or Fluentd to aggregate and analyze logs facilitates faster resolution of issues.

Templates

Several template types define the required functions of a workflow, typically in a container. Template definitions include:

Container — schedules a container. This template’s spec is identical to a Kubernetes container spec, allowing you to define the container in the same way you normally would in Kubernetes. This template type is probably the most popular.
Resource — directly performs operations on a cluster resource. You can use this template for cluster resource requests such as GET, CREATE, APPLY, PATCH, REPLACE, or DELETE.
Script — a convenience wrapper for a container. It is similar to a container spec but with a source: field that lets you define a script in place. You can save the script to a file to be executed. The script produces a result that automatically exports to an Argo variable, like this:
- {{tasks.<NAME>.outputs.result}}
- {{steps.<NAME>.outputs.result}}
Suspend — suspends the execution of a workflow for a specified duration or indefinitely (until you manually resume it). You can resume a suspend template from UI, the API endpoint, or the CLI using the argo resume command.

Additional templates include template invocaters, which you can use to invoke or call other templates and control their execution:

Steps — lets you define workflow tasks as a sequence of steps. The template’s structure contains outer lists that run in sequence and inner lists that run in parallel. You can control execution using a wide range of settings.
DAG — lets you define workflow tasks as a graph of dependencies. The DAG lists all the tasks and specifies the order of their completion (i.e., if a task has dependencies). Argo will run any tasks without dependencies immediately.

Argo Workflows UI

Argo Workflows UI is a web-based user interface for the Argo Workflows engine. It allows you to view completed and live Argo Workflows, and container logs, create and view Argo Cron Workflows, and build new templates.

The UI supports the event-driven automation solution Argo Events, allowing you to use it with Argo Workflows. The UI also offers an event flow page to visualize the connections between sensors and event sources throughout the Argo Project. You can see whether a message was viewed or if a trigger created a workflow.

Argo Workflows UI provides widgets to embed your workflows’ status and progress. You can also embed the latest workflow created by a Cron workflow or template. The updated log viewer allows you to debug artifacts via init and wait containers.

Source: Argo Project

For more information, you can reference the Argo Workflows project GitHub repository, here.

Quick Tutorial: Getting Started with Argo Workflows

This tutorial is based on code and instructions from the Argo Workflows documentation.

1. Install Argo Workflows

Before you begin, ensure you have a Kubernetes cluster and kubectl running on your nodes.

The fastest way to install Argo Workflows is using the quick start manifest. It installs Argo Workflows together with other useful components.

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml

2. Install the Argo CLI

Next, Download the latest Argo CLI:

Download the latest version via curl—the latest URL is available on the releases page
Unzip the binary and make it executable by running chmod
Move the binary to path /usr/local/bin/argo
Test installation by executing argo version

If you did not deploy Argo Workflows using the quick start manifest, you can install it via the CLI with the following command:

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml

3. Get Familiar with the Argo CLI

Here is how to perform common Argo Workflows operations with the Argo CLI:

Operation	CLI Command
Submit a workflow spec to Kubernetes	`argo submit example-spec.yaml`
Display a list of current workflows	`argo list`
Display information about a specific workflow	`argo get example-spec-xxx`
Display logs for a workflow	`argo logs example-spec-xxx`
Delete a workflow	`argo delete example-spec-xxx`

4. Run a Hello World Example

To see Argo Workflows in action, create a simple template that runs the Docker Hub docker/whalesay container image, which echoes “hello world” in the console.

To run the sample image from your shell, use this command:

docker run docker/whalesay cowsay "hello world"

Here is an Argo workflow template that runs this container.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]
      resources:
        limits:
          memory: 32Mi
          cpu: 100m

Let’s review the fields in this template:

kind: Workflow—specifies that this template uses the Workflow spec defined by Argo
generateName—defines the workflow spec name, which will also be the prefix of all pods created by the template
entrypoint: whalesay—specifies that the whalesay template should be the entry point of the Argo workflow. This means that when the workflow spec is executed by Kubernetes, this template will be invoked. In a real workflow, there are multiple templates, and the entry point starts the multi-step process.
resources—limits the resources that the container created by this template will be allowed to use.

For a more extensive tutorial covering additional Argo Workflows operations, see our full Argo Workflow tutorial

Using Argo Workflows with Codefresh

Codefresh now fully incorporates Argo Workflows and Argo Events, including the Codefresh Hub for Argo. This is a central hub for Codefresh platform users and Argo users to consume and share workflow templates.

On a technical level adopting Argo Workflows has the following advantages:

Mature and battle-tested runtime for all pipelines.
Kubernetes native capabilities such as being able to run each step in its own Kubernetes pod.
Offering a set of reusable steps in the form of Workflow Templates
Ability to reuse all existing Artifact integrations instead of spending extra effort to create our own.

The non-technical advantages are just as important:

Organizations familiar with Argo Workflows can adopt Codefresh workflows with minimal effort.
Faster delivery of new features – any new feature shipped by the Argo Workflows project is available to Codefresh users.

See Additional Guides on Key Kubernetes Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of Kubernetes.