Monitoring & managing GitOps Runtimes
Optimize GitOps Runtimes
The Runtimes page displays the provisioned GitOps Runtimes in your account, both Hybrid, and the Hosted Runtime if you have one.
View Runtime components and information in List or Topology view formats to manage and monitor them.
Manage provisioned GitOps Runtimes:
- Add managed clusters to GitOps Runtimes
- Add and manage Git Sources for GitOps Runtimes
- Upgrade GitOps CLI
- Upgrade Hybrid GitOps Runtimes
- Uninstall GitOps Runtimes
Monitor provisioned GitOps Runtimes for security, health, and sync errors:
- (Hybrid and Hosted) View/download logs for Runtimes and for Runtime components
- (Hybrid) Restore provisioned Runtimes
- (Hybrid) Configure browsers to allow access to insecure Runtimes
- (Hybrid) Monitor notifications in the Activity Log
Unless specified otherwise, all options are common to both types of GitOps Runtimes. If an option is valid only for Hybrid GitOps, it is indicated as such.
GitOps Runtime views
View provisioned GitOps Runtimes in List or Topology view formats.
- List view: The default view, displays the list of provisioned Runtimes, the clusters managed by them, and Git Sources associated with them.
- Topology view: Displays a hierarchical view of Runtimes and the clusters managed by them, with health and sync status of each cluster.
List view
The List view is a grid-view of the provisioned Runtimes.
Here is an example of the List view for runtimes.
Here is a description of the information in the List View.
List View Item | Description |
---|---|
Name | The name of the provisioned GitOps Runtime. |
Type | The type of GitOps Runtime provisioned, and can be Hybrid or Hosted. |
Cluster/Namespace | The K8s API server endpoint, as well as the namespace with the cluster. |
Modules | The modules installed based on the type of provisioned Runtime. Hybrid Runtimes include CI amnd CD Ops modules. Hosted runtimes include CD Ops. |
Managed Cluster | The number of managed clusters if any, for the runtime. To view list of managed clusters, select the runtime, and then the Managed Clusters tab. To work with managed clusters, see Adding external clusters to runtimes. |
Version | The version of the runtime currently installed. Update Available! indicates there are later versions of the runtime. To see all the commits to the runtime, mouse over Update Available!, and select View Complete Change Log. |
Last Updated | The most recent update information from the runtime to the Codefresh platform. Updates are sent to the platform typically every few minutes. Longer update intervals may indicate networking issues. |
Sync Status | The health and sync status of the runtime or cluster.
|
Topology view
A hierachical visualization of the provisioned Runtimes. The Topology view makes it easy to identify key information such as versions, health and sync status, for both the provisioned Runtime and the clusters managed by it.
Here is an example of the Topology view for Runtimes.
Here is a description of the information in the Topology view.
Topology View Item | Description |
---|---|
Runtime | ![]() |
Cluster | The local, and managed clusters if any, for the Runtime.
|
Health/Sync status | The health and sync status of the Runtime or cluster.
|
Search and View options |
|
Managing provisioned GitOps Runtimes
- Reset shared configuration repository for GitOps Runtimes
- (Hybrid GitOps) Upgrade provisioned Runtimes
- Uninstall provisioned GitOps Runtimes
- Update Git tokens for Runtimes
Reset shared configuration repository for GitOps Runtimes
Codefresh creates the shared configuration repository when you install the first hybrid or hosted GitOps runtime for your account, and uses it for all runtimes you add to the same account.
If needed, you can reset the location of the shared configuration repository in your account and re-initialize it. For example, when moving from evaluation to production.
Uninstall all the existing runtimes in your account, and then run the reset command. On the next installation, Codefresh re-initializes the shared configuration repo.
Before you begin
Uninstall every runtime in the account
How to
- Run:
cf config --reset-shared-config-repo
(Hybrid GitOps) Upgrade provisioned Runtimes
Upgrade provisioned Hybrid Runtimes to install critical security updates or the latest versions of all components. Upgrade a provisioned Hybrid Runtime by running a silent upgrade or through the GitOps CLI wizard.
If you have managed clusters for Hybrid GitOps, upgrading the Runtime automatically updates runtime components within the managed cluster as well.
When there are security updates, the UI displays the alert, At least one runtime requires a security update. The Version column displays an Update Required! notification.
If you have older Hybrid Runtime versions, upgrade to manually define or create the shared configuration repo for your account. See Shared configuration repo.
Before you begin
For both silent or CLI-wizard based upgrades, make sure you have:
- The latest version of the Codefresh CLI
- A valid Git token with the required scopes
Silent upgrade
-
Pass the mandatory flags in the upgrade command:
cf runtime upgrade <runtime-name> --git-token <git-token> --silent
where:
<git-token>
is a valid Git token with the correct scopes.
CLI wizard-based upgrade
- In the Codefresh UI, on the toolbar, click the Settings icon, expand Runtimes in the sidebar, and select GitOps Runtimes.
- Switch to either the List View or to the Topology View.
- List view:
- Select the Runtime name.
- To see all the commits to the Runtime, in the Version column, mouse over Update Available!, and select View Complete Change Log.
- On the top-right, select Upgrade.
Topology view:
Select the Runtime cluster, and from the panel, select the three dots and then select Upgrade Runtime.
- If you have already installed the GitOps CLI, in the Install Upgrades panel, copy the upgrade command.
- In your terminal, paste the command, and do the following:
- Update the Git token value.
- To manually define the shared configuration repo, add the
--shared-config-repo
flag with the path to the repo.
- Confirm to start the upgrade.
Uninstall provisioned GitOps Runtimes
Uninstall provisioned GitOps Runtimes that are not in use, through a silent uninstall or through the GitOps CLI wizard.
Uninstalling a Runtime removes the Git Sources and managed clusters associated with it.
Before you begin
For both types of uninstalls, make sure you have:
- The latest version of the GitOps CLI
- A valid runtime Git token
- The Kube context from which to uninstall the provisioned Runtime
Silent uninstall
Pass the mandatory flags in the uninstall command:
cf runtime uninstall <runtime-name> --git-token <git-token> --silent
where:
--git-token
is a valid runtime token with the repo
and admin-repo.hook
scopes.
GitOps CLI wizard uninstall
- In the Codefresh UI, on the toolbar, click the Settings icon.
- From Runtimes in the sidebar, select GitOps Runtimes.
- Switch to either the List View or to the Topology View.
- List view: To the right of the runtime to delete, select the three dots and then select Uninstall.
Topology view: Select the Runtime node, and from the panel, select the three dots and then select Uninstall Runtime.
- If you already have the latest version of the GitOps CLI, in the Uninstall Codefresh Runtime panel, copy the uninstall command.
- In your terminal, paste the command, and update the Git token value.
- Select the Kube context from which to uninstall the Runtime, and then confirm the uninstall.
- If you get errors, run the uninstall command again, with the
--force
flag.
Update Git tokens for Runtimes
Provisioned Runtimes require valid Git tokens at all times to authenticate Git actions by you as a user.
These tokens are specific to the user, and the same can be used for multiple runtimes.
There are two different situations when you need to update Git tokens:
- Invalid, revoked, or expired tokens: Codefresh automatically flags Runtimes with such tokens. It is mandatory to update the Git tokens to continue working with the platform.
- Valid tokens: Optional. You may want to update Git tokens, even valid ones, by deleting the existing token and replacing it with a new token.
The methods for updating any Git token are the same regardless of the reason for the update:
- OAuth2 authorization, if your admin has registered an OAuth Application for Codefresh
- Git access token authentication, by generating a personal access token in your Git provider account with the correct scopes
Before you begin
- To authenticate through a Git access token, make sure your token is valid and has the required scopes
How to
- Do one of the following:
- If you see a notification in the Codefresh UI about invalid Runtime tokens, click [Update Token]. The GitOps Runtimes page shows runtimes with invalid tokens prefixed by the key icon. Mouse over shows invalid token.
- To update an existing token, go to GitOps Runtimes.
- Select the GitOps Runtime for which to update the Git token.
- From the context menu at the right, select Update Git Runtime Credentials.
- Do one of the following:
- If your admin has set up OAuth access, click Authorize Access to Git Provider. Go to step 5.
- Alternatively, authenticate with an access token from your Git provider. Go to step 6.
- For OAuth2 authorization:
If the application is not registered, you get an error. Contact your admin for help.
- Enter your credentials, and select Sign In.
- If required, as for example if two-factor authentication is configured, complete the verification.
-
For Git token authentication, expand Advanced authorization options, and then paste the generated token in the Git runtime token field.
-
Click Update Token.
Monitoring GitOps Runtimes
- View/download logs to troubleshoot Runtimes
- (Hybrid GitOps) Restoring provisioned Runtimes
- (Hybrid GitOps) Configure browser to allow insecure Runtimes
- (Hybrid GitOps) View notifications in Activity Log
- (Hybrid GitOps) Troubleshoot health and sync errors for Runtimes
View/download logs to troubleshoot GitOps Runtimes
Logs are available for completed Runtimes, both for the runtime and for individual runtime components. Download log files for offline viewing and analysis, or view online logs for a Runtime component, and download if needed for offline analysis. Online logs support free-text search, search-result navigation, and line-wrap for enhanced readability.
Log files include events from the date of the application launch, with the newest events listed first.
Download logs for GitOps Runtimes
Download the log file for a Runtime. The Runtime log is downloaded as a .tar.gz
file, which contains the individual log files for each runtime component.
- In the Codefresh UI, on the toolbar, click the Settings icon, expand Runtimes in the sidebar, and select GitOps Runtimes.
- If needed, switch to List View, and then select the runtime for which to download logs.
- From the context menu, select Download All Logs.
The log file is downloaded to the Downloads folder or the folder designated for downloads, with the filename,<runtime-name>.tar.gz
. For example,codefreshv2-production2.tar.gz
.
- To view the log files of the individual components, unzip the file.
Here is an example of the folder with the individual logs.
- Open a log file with the text editor of your choice.
View/download logs for Runtime components
View online logs for any Runtime component, and if needed, download the log file for offline viewing and analysis.
Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the application launch to the date and time of download.
- In the Codefresh UI, on the toolbar, click the Settings icon, expand Runtimes in the sidebar, and select GitOps Runtimes.
- If needed, switch to List View, and then select the Runtime.
- Select the Runtime component and then select View Logs.
- Do the following:
- Search by free-text for any string, and click the next and previous buttons to navigate between the search results.
- To switch on line-wrap for readability, click Wrap.
- To download the log, click Download.
The file is downloaded as<component-name>.log
.
(Hybrid GitOps) Restoring provisioned Runtimes
In case of cluster failure, restore the provisioned Hybrid Runtime from the existing runtime installation repository.
For partial or complete cluster failures, you can restore the Runtime to either the failed cluster or to a different cluster.
Restoring the provisioned Runtime reinstalls it, leveraging the resources in the existing Runtime repo.
Restoring the runtime:
- Applies
argo-cd
from the installation manifests in your repo to your cluster - Associates
argo-cd
with the existing installation repo - Applies the Runtime and
argo-cd
secrets to the cluster - Updates the Runtime config map (
<runtime-name>.yaml
in thebootstrap
directory) with the new cluster configuration for these fields:
cluster
ingressClassName
ingressController
ingressHost
Restore a Hybrid Runtime
Reinstall the Hybrid Runtime from the existing installation repository to restore it to the same or a different cluster.
Before you begin
- Have the following information handy:
All values must be the identical to the Runtime to be restored.
- Runtime name
- Repository URL
- Codefresh context
- Kube context: Required if you are restoring to the same cluster
How to
- Run:
cf runtime install --from-repo
- Provide the relevant values when prompted.
-
If you are performing the runtime recovery in a different cluster, verify the ingress resource configuration for
app-proxy
,workflows
, anddefault-git-source
.
If the health status remains asProgressing
, do the following:-
In the Runtime installation repo, check if the
ingress.yaml
files for theapp-proxy
andworkflows
are configured with the correcthost
andingressClassName
:apps/app-proxy/overlays/<runtime-name>/ingress.yaml
apps/workflows/overlays/<runtime-name>/ingress.yaml
-
In the Git Source repository, check the
host
andingressClassName
incdp-default-git-source.ingress.yaml
:resources_<runtime-name>/cdp-default-git-source.ingress.yaml
See the example below.
-
-
If you have managed clusters registered to the hybrid runtime you are restoring, reconnect them.
Run the command and follow the instructions in the wizard:
cf cluster add
-
Verify that you have a registered Git integration:
cf integration git list --runtime <runtime-name>
-
If needed, create a new Git integration:
cf integration git add default --runtime <runtime-name> --provider github --api-url https://api.github.com
Ingress example
This is an example of the ingress.yaml
for workflows
.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/protocol: https
ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/backend-protocol: https
nginx.ingress.kubernetes.io/rewrite-target: /$2
creationTimestamp: null
name: runtime-name-workflows-ingress
namespace: runtime-name
spec:
ingressClassName: nginx
rules:
- host: your-ingress-host.com
http:
paths:
- backend:
service:
name: argo-server
port:
number: 2746
path: /workflows(/|$)(.*)
pathType: ImplementationSpecific
status:
loadBalancer: {}
(Hybrid GitOps) Configure browser to allow insecure Runtimes
If at least one of your Hybrid Runtimes was installed in insecure mode (without an SSL certificate for the ingress controller from a CA), the UI alerts you that At least one runtime was installed in insecure mode.
All you need to do is to configure the browser to trust the URL and receive content.
- Select View Runtimes to the right of the alert.
You are taken to the Runtimes page, where you can see insecure Runtimes tagged as Allow Insecure.
- For every insecure Runtime, select Allow Insecure, and when the browser prompts you to allow access, do as relevant:
- Chrome: Click Advanced and then Proceed to site.
- Firefox: Click Advanced and then Accept the risk and continue.
- Safari: Click Show Certificate, and then select Always allow content from site.
- Edge: Click Advanced, and then select Continue to site(unsafe).
(Hybrid GitOps) View notifications in Activity Log
The Activity Log is a quick way to monitor notifications for Runtime events such as upgrades. A pull-down panel in the Codefresh toolbar, the Activity Log shows ongoing, success, and error notifications, sorted by date, starting with today’s date.
- In the Codefresh UI, on the top-right of the toolbar, select Activity Log.
- To see notifications for provisioned Runtimes, filter by Runtime.
- To see more information on an error, select the + sign.
(Hybrid GitOps) Troubleshoot health and sync errors for Runtimes
The icon with the Runtime in red indicates either health or sync errors.
Health errors
Health errors are generated by Argo CD and by Codefresh for Runtime components.
Sync errors
Runtimes with sync errors display an Out of sync status in Sync Status column. They are related to discrepancies between the desired and actual state of a Runtime component or one of the Git sources associated with the Runtime.
View errors
For both views, select the Runtime, and then select Errors Detected.
Here is an example of health errors for a Runtime.
Related articles
Add Git Sources to GitOps Runtimes
Add external clusters to GitOps Runtimes
Shared configuration repo for GitOps Runtimes