The conversation usually starts with a question like “should we let ArgoCD/Flux/whatever synchronize the actual state automatically whenever the desired state changes in Git?” Truth be told, the question is usually not that elaborated, and it is more like “should I enable the auto-sync feature?” But, I wanted to save you from follow-up questions that help me better understand what that means, so I gave you a more extended and more precise version of the inquiry.
If you are confused and do not understand what’s going on and what that question really means, you are probably not familiar with GitOps deployment tools like Argo CD. If that’s the case, pause reading this article for around 20 minutes. That’s how long it will take you to watch Argo CD: Applying GitOps Principles To Manage Production Environment In Kubernetes. Come back here when you’re done. I’ll wait.
Now, you might think that the answer is something along the lines of “it depends”, but that is not the case. It is straight-forward “you should always enable auto-sync unless you have a justifiable special case which you probably do not.”
The next few minutes are usually spent on people explaining that it is too dangerous to sync automatically. They might say that “someone might push something to a Git repo by mistake, and that something will be automatically propagated to production.” Or it could be along the lines of “our processes are mandating that every change needs to be approved.”
Those questions are usually followed with another round of follow-up questions asking how to sync from CD pipelines, JIRA issues, Excel sheets, or whichever other tools managers might use as a way to feel important and mask their inability to understand one of the most essential tools in our industry. Can you guess which tool that is? No? Let me give you a hint. It is a tool used by every software engineer except those in possession of a time machine that allows them to travel back in time and live in 1999. If you’re still having trouble guessing which tool that is, I’ll give you one more hint. It’s in the title of this article.
That’s right. It’s Git.
Here’s the catch. In most cases, you do not want every commit everyone ever makes to be deployed to production right away.
Some teams are advanced and can allow themselves to push changes directly to the mainline. Those are the teams that have such a good grip on their processes and such a high level of automation that they trust machines to detect issues before they reach production. They have fully automated tests of everything, and they trust them fully. Truth be told, some do not have that. Some are still pushing changes directly to the mainline and deploying each of those changes to production. But those do not matter much for this story since they are usually limited to hobbyists and owners of mom & pop shops.
If you work in a team in charge of a system of a “serious” size, if you are pushing changes directly to the mainline, and if your production rarely “explodes”, all I can say is: “Power to you. I envy the maturity of your team and your processes. Please, please, please send me a job offer.”
Most of us do not push directly to the mainline. We make pull requests (PRs) that initiate pipelines that build artifacts, create releases, run tests, and execute whichever other automated steps we have. We review those PRs, communicate about the potential issues, and do whatever other manual tasks we usually do. Once we are satisfied with the quality of the proposed changes, we merge the PR to the mainline, and that is the signal that a new release should be on its way towards production.
“Why is Viktor talking about all that? Did he grow senile and forgot that he was explaining the pros and cons of doing auto-sync?”
I’m glad you asked.
I’m talking about all that because the answer to the auto-sync question and the need for some to approve deployments lies in pull requests. As for being senile… I’m guessing that senile people are not aware that they are senile, so even if I am, I would not know, or I would have forgotten that I am.
The thing is that enabling auto-sync means that the current state will be changed when the desired state changes. In the context of deployments, the desired state is stored in repositories that describe production, staging, previews, and whichever other environments we might have. Whether something will be deployed right away or after a manual confirmation that the desired state should change depends on whether we create pull requests and which steps are executed in CD pipelines. More importantly, it depends on the permissions we define in Git repositories.
Permissions that define who can push changes to the mainline define who can change the desired state. It does not matter whether auto-sync is enabled in ArgoCD/Flux/what-so-not. The process is as manual or as automated as we want it to be thanks to Git. Pushing a change to the mainline of a repository represents the decision to change the desired state, and it’s up to tools to converge those desires into reality. In other words, keep auto-sync always enabled and control who can do what through Git.
All that means that executing commands like argocd app sync is just silly. It does not matter whether that is done through a CD pipeline, a terminal, or by clicking a button in some UI. Those are just different levels of silliness.
Now, the chances are that you might have a decision-maker in your company that is scared at the prospect of being the person who merges a pull request. There is almost always a manager that insists on having a big red button somewhere else. I do understand that being close to code is scary for some. People might be used to giving approvals from a custom page, JIRA, CD tool of choice, or anywhere else but Git. There is no good reason not to have such a “big red button”. However, the process is still the same. If you want to write a script that will merge a PR whenever that button is pressed, go ahead. Do that. What matters, from the process perspective, is that there is a PR which is merged to the mainline, and when that change modifies the desired state, the tools like Argo CD and Flux will converge the actual into the desired state. For them to do that, auto-sync needs to be enabled.
“Why do we do all that PR nonsense?”
I’m sure that you are not asking yourself that question. If you are, please change the profession. There’s still time for you to become a layer, formula 1 driver, or whatever else you might secretly dream of becoming.
The point is that everything should be defined as code, that code is stored in Git, and that the decisions are made by merging code from one place to another. Today, we do that through PRs.
As a result, we have full traceability of the desired state. We know what it is today and what it was yesterday. We know who did what and why through pull requests and the associated commits, comments, and issues, and not through arbitrary ad-hoc executions of argocd app sync or similar commands. It does not matter whether they were executed by us (humans) or by machines (e.g., CD pipelines).
The additional benefit, besides the obvious reasons, is security. Push mechanisms are insecure. If you can access production, you are a liability. Even if I trust you fully, the fact that you can do it means that there is a potential opening for others to do it as well. Similarly, we shouldn’t give that access to the tools either. Why would you, for example, allow some third-party tool, potentially a SaaS offering, to access your production if there is no justifiable benefit to do so. If ArgoCD/Flux/whatever-else-is-managing-the-actual-state can pull the information from Git, there is no need to enable incoming traffic to the cluster. That is what the pull mechanism gives us. It allows us to remove all incoming traffic since the tool in charge of deployments is running inside that cluster, and it only has to pull data.
I just realized that I could continue like this for a long time, and the original intention was to be brief. So, consider what’s coming next as me getting back on track.
Should you enable auto-sync in GitOps deployment tools like Argo CD, Flux, and others? The answer is a huge yes. Does that mean that releases are deployed to production without any control? That depends on your Git permissions rather than whether auto-sync is enabled.
If you are interested in GitOps and continuous delivery, please try Codefresh. You’ll get free unlimited builds for life.