Trials and Tribulations of K8s Debugging

Trials and Tribulations of K8s Debugging

4 min read

Editors Note: This is a guest blog post provided by Rookout. This very topic was also featured as a webinar on #CodefreshLive. Scroll down to view the full webinar.

If you work with microservices, it’s likely that you rely on Kubernetes to run code. But when it comes to debugging, this technological match-made-in-heaven goes suddenly south.

Kubernetes recommends running your microservices, together with debugging tools, within a remote shell on a running container. This approach, however, is costly, cumbersome and entirely foreign to most devs working today.

Yet, for developers who prefer local debugging – which is to say, almost all devs – the interdependencies of microservice architectures makes microservice-based apps exceptionally difficult to replicate or simulate on a local machine.

So what’s a developer to do when s/he needs to debug?

The good news is that many solutions that simulate microservice environments on local machines are already available. However, none is the magic bullet that makes microservice debugging easy to do.

Let’s review the solutions that are out there and see how they work – or do not.

Local debugging solutions

  • Automation scripts enable devs to run microservices on their own machines. So far, so good, right? But scripts tend to break since configurations must be tightly controlled and branches must remain aligned. As a result, this is a high-maintenance approach that is frustrating for devs.
  • Hotel is like a local process manager for running microservices. This open-source project enables devs to start and stop services and view all logs in a browser tab. Like automation scripts, however, smooth operation of Hotel depends on controlled configurations and tightly aligned branches. And devs must invest time and effort in learning a new tool.
  • Docker Compose is a tool for defining and running multi-container Docker apps. As such, architectural changes must be recorded in the Docker Compose YAML. While configuration is more dev-friendly than for automation scripts, advance Kubernetes configurations can be difficult to replicate. In addition, logs from all microservices are written in one location, making it difficult to match each microservice with the log data it has generated.
  • Minikube, an official CNCF tool for spinning up Kubernetes instances on local machines, should, by all rights be the best solution. So it’s disappointing that K8S configurations generally do not work in Minikube without some adjustment. Developers using Minikube also need to know how to change Kubernetes configurations, a skill that is foreign and frightening for many devs.

Remote debugging solutions

Given the decidedly mixed success of these local debugging approaches, perhaps it’s time to explore options for debugging Kubernetes remotely, daunting as the concept may seem.

Kubernetes orchestration mechanisms and optimization methodologies are great for microservices development and production, but they make debugging tough since access to pods is highly unstable. Using an SSH command to run debugging tools on a pod is likely to trigger K8S to shut down the tools before you get the data you seek.

With that warning in mind, let’s have a look at the various methods you can use to debug microservices in Kubernetes.

  • Adding logs and redeploying is the classic approach. Unfortunately, it is also expensive and time-consuming – even if you get enough of the right data to get insight into the issue’s root cause (which is by no means always the case).
  • Attaching to a process is challenging you will have to make sure the process id namespace between the debugger and the application are shared. And then K8S may kill your pod at any moment, causing contact with the debugger to be lost.
  • Redirecting traffic from the cluster to the dev machine can help you recreate issues. However, it is insecure and can easily overtax local computers if a great deal of data is involved. Installation of a DaemonSet on each node to mount the container runtime socket may be required to see all process on all nodes. And redirecting traffic exposes data to the dangers of the internet
  • Service meshes, such as Istio and Linkerd enable you to track the interactions within a network of microservices, as well as inbound and outbound traffic, without changing code. You can see the entire flow of a request through the microservices stack, thanks to distributed tracing capabilities, so you can pinpoint requests or microservices that might be problematic. However, additional tools are necessary to find the root cause of issues, since service meshes can tell you that there is a problem, but not why that problem occurs. 
  • Adding logs at runtime by adding an SDK to your code is the easiest way to deploy to Kubernetes architecture. It allows you to add logs on the fly without writing more code and redeploying, so you can get data from code in real-time. 

Live datapoint collection

While some solutions for adding logs at runtime, like Stackdriver, are limited to specific cloud services, Rookout enables rapid debugging for dev, staging and production environments in any cloud. It supports a wide range of languages, including Node.JS, JVM-based languages and Python-based applications for both PyPy and CPython interpreters (with more to come!).

Rookout requires no additional code, restarts or redeployment. Add non-breaking breakpoints, as you would with a regular debugger, to get data instantly. Even as data is pipelined on the fly to your choice of destination, the app continues to operate without interruption. 

 Watch the #CodefreshLive webinar including a live demo here.

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

Build your GitOps skills and credibility today with a GitOps Certification.

Get GitOps Certified

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments