Add Kubernetes Cluster

How to connect your Kubernetes cluster to the Codefresh dashboard

Codefresh offers its own Kubernetes dashboard that allows you to inspect the services and namespaces in your cluster. To active this dashboard you need to connect your cluster to your Codefresh account first.

Start by going into your Account Configuration, by clicking on Account Settings on the left sidebar. On the first section called Integrations click the Configure button next to Kubernetes.

Codefresh integrations

Codefresh integrations

In the Kubernetes integration window, you will be able to add a cluster from known providers such as Google, Azure, Amazon etc. You can also add any generic kubernetes cluster by manually entering your cluster settings.

Adding GKE Cluster

Adding a cluster in GKE can be done by clicking the Add cluster button under Google Cloud Provider and selecting the desired project and cluster.

If this is your first time you’ll be prompted to authenticate using your google credentials, make sure you’re doing so with a user that have access to your GKE projects.

Make sure that your cluster has basic authentication enabled. You can change this setting after cluster creation by editing and changing the setting from the drop-down list.

Enabling GKE basic authentication

Enabling GKE basic authentication

If you are a new customer of Google Cloud, you are also eligible to receive a Codefresh offer to get up to $500 in Google credits. As soon at the GKE integration is complete within Codefresh, you will get an email with extra details on how to claim your credits.

Follow the link in the email to fill in an application for the free credits. Once Google approves the application (usually within 1-2 days) your credits will be available to your account. Make sure to check your spam folder for that email.

Adding AKS cluster

To add an Azure cluster, select Azure AKS from the drop-down menu. Click the Authenticate button and enter your Azure credentials. You will see a description of all permissions that Codefresh needs in order to access your cluster. Accept them and Codefresh will connect to Azure to get the cluster information.

If you experience difficulties at this point try logging into Azure first in your browser before clicking the authenticate button. Also make sure that you are using an organizational/company Azure account and not a personal one. We are currently working with Microsoft to improve this integration.

If everything is ready you will see a dialog that allows you to select your Azure subscription and the cluster name that you wish to use.

Selecting the Azure cluster

Selecting the Azure cluster

Codefresh will query the cluster and show its nodes. You are now ready to deploy to Azure kubernetes.

If you wish for any reason to revoke the granted access from the Azure side, visit https://account.activedirectory.windowsazure.com/r#/applications and remove “Codefresh” from the list.

Adding EKS Cluster

To add an Amazon EKS cluster, you must first create a service account and obtain a token used to manage the integration.

The official Amazon-provided guide on EKS can be found here.

In order to use your cluster locally with kubectl, you must first install the heptio-authenticator-aws binary. Your version of kubectl must also be 1.10+ for this to work.

Next, create a kubeconfig file, such as ~/.kube/eks, replacing <endpoint-url>, <base64-encoded-ca-cert>, and <cluster-name> with information on your EKS cluster obtained in the AWS console:

apiVersion: v1
clusters:
- cluster:
    server: <endpoint-url>
    certificate-authority-data: <base64-encoded-ca-cert>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "<cluster-name>"

Then, in an environment that has access to your AWS account, run the following command to create an admin user service account and necessary role binding:

cat <<EOF | kubectl --kubeconfig="$HOME/.kube/eks" apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

Finally, use the following command to obtain the service account token:

kubectl --kubeconfig="$HOME/.kube/eks" -n="kube-system" get secret \
  $(kubectl --kubeconfig="$HOME/.kube/eks" -n="kube-system" get secret | \
    grep admin-user | awk '{print $1}') -o jsonpath="{.data.token}"

Once you have this token, follow the steps in the section below, using this token for item #4.

Adding any other cluster type (not dependent on any provider)

Go to your Account Configuration, by clicking on Account Settings on the left sidebar. On the first section called Integrations click the Configure button next to Kubernetes.

In order to add any other type of cluster, outside of GKE, use Custom Providers

Adding a custom cluster in Codefresh

Adding a custom K8s cluster in Codefresh

The integration between Codefresh and your Kubernetes cluster is API based and relies on a Kubernetes service account of your choosing that will be used to manage the integration.

The configurations you’ll be required to add are:

  1. Name - Any name of your choosing, that will represent your cluster context in Codefresh.
  1. Host - The full URL of the Kubernetes API endpoints including protocol and port
  1. Certificate - The Kubernetes service account certificate used for the integration with Codefresh (base64 encoded)
  1. Token - The Kubernetes service account token used for the integration with Codefresh (base64 encoded)

Adding a custom cluster in Codefresh - details

Adding a custom cluster in Codefresh - details

In the section below we’ll provide you with easy instructions how to get all your cluster configurations in order to add it to Codefresh.

Get cluster configuration manually

Copy and paste the following commands into your local shell, then save the outputs and paste them into the Codefresh fields. The commands rely on kubectl so make sure it is configured correctly against your cluster.

More than one cluster in kubeconfig?

Before starting, make sure that your local context is the one you would like to add to Codefresh.
Switch to the desired context before continue

Host IP

export CURRENT_CONTEXT=$(kubectl config current-context) && export CURRENT_CLUSTER=$(kubectl config view -o go-template="{{\$curr_context := \"$CURRENT_CONTEXT\" }}{{range .contexts}}{{if eq .name \$curr_context}}{{.context.cluster}}{{end}}{{end}}") && echo $(kubectl config view -o go-template="{{\$cluster_context := \"$CURRENT_CLUSTER\"}}{{range .clusters}}{{if eq .name \$cluster_context}}{{.cluster.server}}{{end}}{{end}}")

Certificate

echo $(kubectl get secret -o go-template='{{index .data "ca.crt" }}' $(kubectl get sa default -o go-template="{{range .secrets}}{{.name}}{{end}}"))

Token

echo $(kubectl get secret -o go-template='{{index .data "token" }}' $(kubectl get sa default -o go-template="{{range .secrets}}{{.name}}{{end}}"))
Note

In the instructions above, we’re referring for a service account named ‘default’ in regards to the certificate and token. You can provide any service account configurations you may have on any namespace, as long as it has the correct permissions. The cluster actions you’ll be limited to in Codefresh are based on the Kubernetes service account permissions you set in Kubernetes RBAC.

The minimum permissions Codefresh needs to work with the cluster are the following:

codefresh-role.yml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: codefresh-role
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["list", "watch", "get"] 

Once the cluster been added successfully you can go to the Kubernetes tab to start working with the services of your cluster.

Adding a Rancher cluster

Rancher clusters are currently supported as generic clusters. Rancher clusters have a specific authentication configuration (the details are here: https://rancher.com/kubernetes-authentication-in-rancher-and-rbac).

Authentication using a token of a Kubernetes Service Account, which is usually used by Codefresh, doesn’t work with Rancher clusters. Also, Rancher doesn’t do proper TLS termination out-of-the-box for Kubernetes clusters hosted on it, so one needs to configure a load balancer for that purpose.

In summary, the following conditions should be met in order to add the cluster, hosted on Rancher to Codefresh:

  1. The token should be taken from the kubeconfig provided by Rancher and it has to be encoded with base64 before putting it into Codefresh. Be careful with the ‘\n’ characters when encoding. The command for Linux is: echo <rancher_token> | tr -d '\n' | base64 | tr -d '\n'
  2. The CA certificate should be the CA of the Load Balancer standing in front of Rancher
  3. The hostname and port should be corresponding to your Load Balancer

Getting the Rancher token

Getting the Rancher token

Once you have all the information follow the instructions for adding a generic Kubernetes cluster in Codefresh as described in the previous section.

Troubleshooting cluster addition

After adding your cluster configurations and in case the test fails, click “Save” to get the error message back.

click-save.png

Error: Cannot list namespaces

Add Cluster Error

Failed to add cluster: namespaces is forbidden: User "system:serviceaccount:default:default" cannot list namespaces at the cluster scope

The service account used for the integration doesn’t have the minimal permissions required. To fix this add a service account that have the required permissions.

The easiest way to do this is to create a cluster binding role between the default service account and cluster-admin role:

Create cluster binding with admin permissions

kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default

Kubernetes cluster - using an external reverse proxy (edge case)

In case you’re using an external reverse proxy to manage inbound traffic to your Kubernetes API, please read this article to make sure your certificate setup are managed correctly in order to add your cluster successfully to Codefresh.

Multiple CAs in certificate chain

If you have more than one CA in your certification chain, you need to provide Codefresh with a Certificate bundle (a file that containers the intermediate CAs as well)

The steps needed are:

  1. Get your CA bundle file and the Kubernetes API server certificate file and run the following to check the validity of the certificate: openssl verify -verbose -CAfile ca_bundle.pem k8_server_cert
  2. If the check above passes fine, go on and run the following on your CA bundle file: base64 ca_budle.pem | tr -d '\n'
  3. Copy the output string (be careful when copying) and check whether you have copied it correctly: openssl x509 -text -in <(echo <copied_string> | base64 -d) - you should see the contents of your CA budle file
  4. Put the copied string into the Codefresh Kubernetes integration form and test the connection.