Hybrid runtime requirements

The requirements listed are the minimum requirements to provision hybrid runtimes in the Codefresh platform.

Hosted runtimes are managed by Codefresh. To provision a hosted runtime as part of Hosted GitOps setup, see Provision a hosted runtime in Set up a hosted (Hosted GitOps) environment.

In the documentation, Kubernetes and K8s are used interchangeably.


Minimum requirements

Item Requirement
Kubernetes cluster Server version 1.18 and higher, without Argo Project components.
Tip: To check the server version, run:
kubectl version --short.
Ingress controller Configured on Kubernetes cluster and exposed from the cluster.
Supported and tested ingress controllers include:
Node requirements
  • Memory: 5000 MB
  • CPU: 2
Cluster permissions Cluster admin permissions
Git providers
  • GitHub
Git access tokens Runtime Git token:
  • Valid expiration date
  • Scopes: repo and admin-repo.hook
Personal access Git token:
  • Valid expiration date
  • Scopes: repo


Ambassador ingress configuration

For detailed configuration information, see the Ambassador ingress controller documentation.

This section lists the specific configuration requirements for Codefresh to be completed before installing the hybrid runtime.

  • Valid external IP address
  • Valid TLS certificate
  • TCP support

Valid external IP address

Run kubectl get svc -A to get a list of services and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.

Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.



AWS ALB ingress configuration

For detailed configuration information, see the ALB AWS ingress controller documentation.

This table lists the specific configuration requirements for Codefresh.

What to configure When to configure
Valid external IP address Before installing hybrid runtime
Valid TLS certificate  
TCP support  
Controller configuration]  
Alias DNS record in route53 to load balancer After installing hybrid runtime
(Optional) Git integration registration  

Valid external IP address

Run kubectl get svc -A to get a list of services and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.


Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.


Controller configuration

In the ingress resource file, verify that spec.controller is configured as ingress.k8s.aws/alb.

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: alb
spec:
  controller: ingress.k8s.aws/alb

Create an alias to load balancer in route53

The alias must be configured after installing the hybrid runtime.

  1. Make sure a DNS record is available in the correct hosted zone.
  2. After hybrid runtime installation, in Amazon Route 53, create an alias to route traffic to the load balancer that is automatically created during the installation:
    • Record name: Enter the same record name used in the installation.
    • Toggle Alias to ON.
    • From the Route traffic to list, select Alias to Application and Classic Load Balancer.
    • From the list of Regions, select the region. For example, US East.
    • From the list of load balancers, select the load balancer that was created during installation.

For more information, see Creating records by using the Amazon Route 53 console.

Route 53 record settings for AWS ALB

Route 53 record settings for AWS ALB

(Optional) Git integration registration

If the installation failed, as can happen if the DNS record was not created within the timeframe, manually create and register Git integrations using these commands:
cf integration git add default --runtime <RUNTIME-NAME> --api-url <API-URL>
cf integration git register default --runtime <RUNTIME-NAME> --token <RUNTIME-AUTHENTICATION-TOKEN>



Istio ingress configuration

For detailed configuration information, see Istio ingress controller documentation.

The table below lists the specific configuration requirements for Codefresh.

What to configure When to configure
Valid external IP address Before installing hybrid runtime
Valid TLS certificate  
TCP support  
Cluster routing service After installing hybrid runtime

Valid external IP address

Run kubectl get svc -A to get a list of services and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.


Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.


Cluster routing service

The cluster routing service must be configured after installing the hybrid runtime.

Configure the VirtualService to route traffic to the app-proxy and webhook services, as in the examples below.


VirtualService example for app-proxy:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: test-runtime3 # replace with your runtime name
  name: cap-app-proxy 
spec:
  hosts:
    - my.support.cf-cd.com # replace with your host name
  gateways:
    - my-gateway
  http:
    - match:
      - uri:
          prefix: /app-proxy 
      route:
      - destination:
          host: cap-app-proxy 
          port:
            number: 3017

VirtualService example for webhook:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: test-runtime3 # replace with your runtime name
  name: csdp-default-git-source
spec:
  hosts:
    - my.support.cf-cd.com # replace with your host name
  gateways:
    - my-gateway
  http:
    - match:
      - uri:
          prefix: /webhooks/test-runtime3/push-github # replace `test-runtime3` with your runtime name
      route:
      - destination:
          host: push-github-eventsource-svc 
          port:
            number: 80


NGINX Enterprise ingress configuration

For detailed configuration information, see NGINX ingress controller documentation.

The table below lists the specific configuration requirements for Codefresh.

What to configure When to configure
Verify valid external IP address Before installing hybrid runtime
Valid TLS certificate  
TCP support  
NGINX Ingress: Enable report status to cluster  
NGINX Ingress Operator: Enable report status to cluster  
Patch certificate secret After installing hybrid runtime

Valid external IP address

Run kubectl get svc -A to get a list of services and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.


Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.


NGINX Ingress: Enable report status to cluster

If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation.

  • Pass --report-ingress-status to deployment.
spec:                                                                                                                                                                 
  containers: 
    - args:                                                                                                                                              
      - --report-ingress-status

NGINX Ingress Operator: Enable report status to cluster

If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation.

  1. Add this to the Nginxingresscontrollers resource file:

    ...
    spec:
      reportIngressStatus:
        enable: true
    ...
    
  2. Make sure you have a certificate secret in the same namespace as the runtime. Copy an existing secret if you don’t have one.
    You will need to add this to the ingress-master when you have completed runtime installation.


Patch certificate secret

The certificate secret must be configured after installing the hybrid runtime.

Patch the certificate secret in spec.tls of the ingress-master resource.
The secret must be in the same namespace as the runtime.

  1. Go to the runtime namespace with the NGINX ingress controller.
  2. In ingress-master, add to spec.tls:

     tls:                                                                                                                                                                    
      - hosts:                                                                                                                                                                
      - <host_name>                                                                                             
      secretName: <secret_name>
    


NGINX Community version ingress configuration

Codefresh has been tested with and supports implementations of the major providers. For your convenience, we have provided configuration instructions, both for supported and untested providers in Provider-specific configuration.

This section lists the specific configuration requirements for Codefresh to be completed before installing the hybrid runtime.

  • Verify valid external IP address
  • Valid TLS certificate
  • TCP support

Valid external IP address

Run kubectl get svc -A to get a list of services, and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.


Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.

Here’s an example of TCP configuration for NGINX Community on AWS.
Verify that the ingress-nginx-controller service manifest has either of the following annotations:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
OR
service.beta.kubernetes.io/aws-load-balancer-type: nlb


Provider-specific configuration

The instructions are valid for k8s.io/ingress-nginx, the community version of NGINX.

AWS
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for AWS.
Azure (AKS)
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for AKS.
Bare Metal Clusters
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
Bare-metal clusters often have additional considerations. See Bare-metal ingress-nginx considerations.
Digital Ocean
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/do/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for Digital Ocean.
Docker Desktop
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for Docker Desktop.
Note: By default, Docker Desktop services will provision with localhost as their external address. Triggers in delivery pipelines cannot reach this instance unless they originate from the same machine where Docker Desktop is being used.
Exoscale
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for Exoscale.
Google (GKE)
Add firewall rules
GKE by default limits outbound requests from nodes. For the runtime to communicate with the control-plane in Codefresh, add a firewall-specific rule.
  1. Find your cluster's network:
    gcloud container clusters describe [CLUSTER_NAME] --format=get"(network)"
  2. Get the Cluster IPV4 CIDR:
    gcloud container clusters describe [CLUSTER_NAME] --format=get"(clusterIpv4Cidr)"
  3. Replace the `[CLUSTER_NAME]`, `[NETWORK]`, and `[CLUSTER_IPV4_CIDR]`, with the relevant values:
    gcloud compute firewall-rules create "[CLUSTER_NAME]-to-all-vms-on-network"
    --network="[NETWORK]" \
    --source-ranges="[CLUSTER_IPV4_CIDR]" \
    --allow=tcp,udp,icmp,esp,ah,sctp

Use ingress-nginx
  1. Create a `cluster-admin` role binding:
    kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin \
    --user $(gcloud config get-value account)
  2. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
  3. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
We recommend reviewing the provider-specific documentation for GKE.
MicroK8s
  1. Install using Microk8s addon system:
    microk8s enable ingress
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
MicroK8s has not been tested with Codefresh, and may require additional configuration. For details, see Ingress addon documentation.
MiniKube
  1. Install using MiniKube addon system:
    minikube addons enable ingress
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
MiniKube has not been tested with Codefresh, and may require additional configuration. For details, see Ingress addon documentation.
Oracle Cloud Infrastructure
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for Oracle Cloud.
Scaleway
  1. Apply:
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/scw/deploy.yaml
  2. Verify a valid external address exists:
    kubectl get svc ingress-nginx-controller -n ingress-nginx
For additional configuration options, see ingress-nginx documentation for Scaleway.


Traefik ingress configuration

For detailed configuration information, see Traefik ingress controller documentation.

The table below lists the specific configuration requirements for Codefresh.

What to configure When to configure
Valid external IP address Before installing hybrid runtime
Valid SSL certificate  
TCP support  
Enable report status to cluster  

Valid external IP address

Run kubectl get svc -A to get a list of services and verify that the EXTERNAL-IP column for your ingress controller shows a valid hostname.


Valid TLS certificate

For secure runtime installation, the ingress controller must have a valid TLS certificate.

Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate.


TCP support

Configure the ingress controller to handle TCP requests.


Enable report status to cluster

By default, the Traefik ingress controller is not configured to report its status to the cluster. If not configured, Argo’s health check reports the health status as “progressing”, resulting in a timeout error during installation.

To enable reporting its status, add publishedService to providers.kubernetesIngress.ingressEndpoint.

The value must be in the format "<namespace>/<service-name>", where:
<service-name> is the Traefik service from which to copy the status

...
providers:
  kubernetesIngress:
    ingressEndpoint:
      publishedService: "<namespace>/<traefik-service>" # Example, "codefresh/traefik-default" 
...

Hybrid runtime installation flags
Install hybrid runtimes