Everything You Ever Wanted to Know About Deletion and Argo CD Finalizers but Were Afraid to Ask

Accidentally deleting an application is bad, but accidentally deleting all of them is a lot worse. 

In this post, I show you how to avoid issues like the following:

• A user deletes a troublesome resource expecting it to get recreated. Instead, this deletion triggers the deletion of other resources. Ouch!
• The ApplicationSet controller suddenly deletes all applications without warning.
• An admin upgrading Argo CD deletes some components that seem to be stuck. In the process, they accidentally delete every application managed by the Argo CD instance.
• After accidentally deleting applications, an admin deletes Argo CD components. This leaves resources in a deletion limbo that prevents their recreation. 

Argo CD finalizers is a built-in Argo CD feature that confuses many new users and can even result in accidental deletion if you’re not careful. If you see a resource managed by Argo CD that was supposed to be deleted but isn’t, then you can almost certainly blame a finalizer. On the other hand, if you wiped out your application by mistake, you probably missed a finalizer.

In Argo CD, there are multiple ways to wipe out your application and the resources it deploys. This can cause your application to experience an outage, or worse, issues blocking recovery. 

In this post, you learn about several use cases for finalizers. I also share possible scenarios of how and when your application can get destroyed. I review the application and application set controllers, and the various ways of deleting apps from the UI, Git, and the cluster.

How Argo CD resources work together

First, let’s look at the overall Argo CD architecture. This diagram represents the components that manage Argo CD application custom resources and ApplicationSet custom resources.

At the heart of the system, we have the application controller deciding all aspects of an application and where and how it’s deployed. The ApplicationSet controller can automatically generate individual applications but is otherwise not involved in any deployment operation. The Argo CD repository server is responsible for Git communication and retrieving and rendering manifests.

How to use finalizers

Finalizers aren’t unique to Argo CD. In Kubernetes, finalizers let you delegate responsibility to a controller and prevent resource deletion. You can annotate a finalizer on any Argo CD application.

metadata:
  finalizers:
    # The default behavior is foreground cascading deletion
    - resources-finalizer.argocd.argoproj.io
    # Alternatively, you can use background cascading deletion
    # - resources-finalizer.argocd.argoproj.io/background

As a basic rule, be careful where you place finalizers. The presence of a finalizer instructs Argo CD that when a deletion occurs, it needs to delete several other resources as well. Make sure this is what you want for the resource that gets annotated.

When deleting an application with this finalizer, the ArgoCD application controller will perform a cascading delete of the application’s resources. Adding the finalizer enables cascading deletes when implementing the App of Apps pattern. The default propagation policy for cascading deletion is foreground cascading deletion. Argo CD performs background cascading deletion when resources-finalizer.argocd.argoproj.io/background is set.

When you create an application from the UI, you can set up a deletion finalizer. This ensures that cascade resource deletion occurs after the application CR gets deleted.

How deletion works for ApplicationSets

We’ve seen how finalizers work for individual applications. What happens with ApplicationSets? The ApplicationSet controller is a tool that generates applications and connects them with owner references (Kubernetes owner references). By default, the ApplicationSet controller generates an application with a resource finalizer already active. Based on the application deletion flow, if a finalizer exists in the application, the Argo CD controller deletes all child resources in a cascading manner. For extra safety, the application set supports the .syncPolicy.preserveResourcesOnDeletion flag. This prevents the ApplicationSet controller from setting up a finalizer during application generation.

The end result when an ApplicationSet gets deleted is the following (in rough order):

1. The ApplicationSet resource itself gets deleted.
2. Any Application resources created from this ApplicationSet (as identified by owner reference) get deleted.
3. Any deployed resources (Deployments, Services, ConfigMaps, etc.) on the managed cluster that were created from that Application resource (by Argo CD) get deleted.

Argo CD handles this deletion via the deletion finalizer. If you want to preserve deployed resources, set .syncPolicy.preserveResourcesOnDeletion to true in the ApplicationSet.

Use case 1 – Removing an application

Let’s examine the base case first. Finalizers are useful because they let you delete an application and all its associated resources.

Deleting an application from the UI

This flow describes how Argo CD handles application deletion from the UI. The same process occurs when you delete an application from Git. The only difference is that you can’t override the finalizer from the UI. It always uses the one defined in Git.

Deleting an application from Git

To delete an application from Git, you must enable the prune option.

This flow shows application deletion from Git.

Note: To delete an application from Git, enable the prune option.

Use case 2 – Troubleshooting application deletion

Let’s look at the worst-case scenario. Someone deleted a set of Kubernetes resources by mistake. How can you detect, verify, and audit this scenario?

The first line of defense is to check the logs of the individual Argo CD components (and ideally omit events when deletions happen).

Logs for ArgoCD Server:

1. “Deleted application” – This log indicates the user manually initiated app deletion.

Logs for ArgoCD Application Controller:

• “Deleting resources” – This means there was a request to delete the application in Kubernetes, and the ArgoCD controller is deleting all child resources.
• “Objects remaining for deletion” – Deletion of child resources is in progress.
• “Deleting application’s resources with %s propagation policy” – This means there was a request to delete the application in Kubernetes, and it shows the propagation policy.

Set up your monitoring system to alert you when deletions occur. This way, you have a history of deletion events and know in advance if a deletion was a mistake.

Use case 3 – Renaming an application

Contrary to popular belief, you cannot modify the Argo CD application name. If you do, ArgoCD will delete the existing application and create a new one. This results in downtime.

It also means that if you have a finalizer on the existing application, Argo CD will delete all its child resources. These child resources might be costly to recreate, so again make sure the finalizer exists by design and not by mistake.

In practice, you should not rename your Argo CD applications. In the next use case, I explain the scenario of moving an Argo CD application somewhere else.Similarly, renaming an ApplicationSet also forces Argo CD to recreate it, which can cause destructive behavior and downtime. It’s worth mentioning that Codefresh supports a non-destructive method to rename applicationsets.

Use case 4 – Moving or renaming an Argo CD application

When managing applications, it’s crucial to handle migrations or deletions safely to prevent disruptions. This is a very popular scenario when updating Argo CD versions. Several companies do the following when a new version of Argo CD gets released:

1. Clone the main Kubernetes cluster X to a brand new cluster Y.
2. Install the new version of Argo CD on cluster Y.
3. Move Argo CD applications from cluster X to cluster Y.
4. Delete cluster X (the old Argo CD instance).

You can also follow a similar process for renaming applications.

Decoupling applications from Kubernetes resources

The key point here is understanding that a set of Kubernetes applications can continue to exist and work normally on the cluster without having a parent Argo CD entity to group them. So the plan when you want to migrate/move an Argo CD application without downtime is the following:

1. Create an Argo CD application inside the cluster where you want to move the application, and point it to the same destination cluster. 
2. The new Argo CD application will now reference the existing resources.
3. Ensure the existing Argo CD application doesn’t have an Argo CD resource finalizer. 
4. Delete the Argo CD application.
5. The Kubernetes resources will remain in the cluster without downtime.

Before deleting an application, ensure it doesn’t have an ArgoCD resource finalizer. Finalizers manage the deletion lifecycle of resources, preventing accidental deletion while resources are still being processed.

Moving applications between instances

Scenario: Migrating an application named “gb” to another Argo CD instance

Checking for finalizers

First, you need to check if there are any finalizers under the metadata of your application. Run the following command inside the cluster where Argo CD is located:

kubectl get app gb -o yaml -n <namespace>

This command retrieves the YAML configuration of the gb application. You can find the finalizers field under the metadata section.

Handling finalizers

If a finalizer exists, it means that after you remove or replace an Argo CD application, the Argo CD controller will delete all resources managed by the gb application. To prevent this, you can either:

• Change the prune propagation policy via the Argo CD UI:
  • Go to the application details in the Argo CD UI.
  • Press the edit button.
  • Change the prune propagation policy to orphan.
• Remove the finalizers using kubectl:
  • Execute the following command to edit the application metadata:
    kubectl edit app gb -n <namespace>
  • In the editor, locate and delete the metadata.finalizers field, then save and exit.

Migrating applications

While it’s easier to migrate applications if they’re persisted in Git, you can also migrate them manually if needed. Simply copy the application manifest from one Argo CD instance and apply it to another instance.

Export and clean up the application

Export the application YAML:

1. kubectl get app gb -o yaml > app.yaml.
2. Ensure the finalizer does not exist in the exported YAML.

Delete the application:

1. kubectl delete app gb -n <namespace> --cascade=orphan.
2. Verify resources: After deletion, ensure that your resources still exist on the destination cluster.

Apply the application to the new instance

Switch Kubernetes context:

kubectl config use-context <new-context>

Apply the application manifest:

kubectl apply -f app.yaml -n <namespace>

If your application manifest is in Git, you need to make sure finalizers are not set. After that, you can safely remove the Argo CD application custom resource and add it to another Argo CD instance.

Another option is to apply the application manifest on the new Argo CD instance first, and then delete it from the old instance.

Conclusion

We hope this guide has addressed the mysteries of finalizers and their use. You should now know when to use finalizers and how to avoid tricky situations.

If the worst-case scenario happens, you should always have all your resources stored in Git. This means both the Kubernetes resources and the Argo CD manifests (including applications sets). At all times, you should be able to recreate any Argo CD application from Git in a matter of seconds, even if you deleted it by mistake.