The ultimate guide for local development on Kubernetes: Draft vs Skaffold vs Garden.io

The ultimate guide for local development on Kubernetes: Draft vs Skaffold vs Garden.io

17 min read

In the previous articles of this series, we have seen local Kubernetes installations designed specifically for developers. We have looked at all the major players such as Minikube, Docker-for-desktop, and Microk8s.

However, we have conveniently left out the details on how you can use a local Kubernetes cluster as a developer to test a feature on your workstation before actually committing anything (i.e. before the CI/CD pipeline takes over).

By automating the local development workflow, we can significantly reduce the deployment and testing phases and provide a quick feedback loop which is always crucial for developer productivity.

Of course, you could always run manual kubectl commands to deploy to your local cluster. But this process becomes quickly cumbersome and can be problematic in cases of microservices where you might have complex dependencies between the deployed applications.

Today, there are many available tools designed for Kubernetes deployments specifically for developers. Their primary goal is to help you make the feedback loop as fast as possible (code-deploy-refactor) and they can be used with any kind of cluster (remote or local) making them the perfect solution for developers who wish to develop Kubernetes enabled applications.

In this article, we are reviewing three such development tools that are helping developers to work with local Kubernetes clusters. These tools are Draft, Skaffold and Garden.

Note: After we published this article, tilt.dev appeared which is another alternative on this space. See the tilt.dev blog post for more details.

Local Kubernetes development with Draft

Draft is a project developed by Microsoft, in order to assist developers to build applications that can be deployed to a Kubernetes cluster (either local or remote). Draft targets the “inner loop” of development workflow, meaning it helps developers deploy their feature to a cluster, without committing anything first to source control.

Draft can also automatically discover the type of code that you are working on and help you generate all needed configuration files. You can use Draft to create both the Dockerfile and the Kubernetes manifests for an existing application.

The second important Draft feature is the automatic deployments of your application into your a Kubernetes cluster. This means that code changes are automatically synchronized with the cluster deployment.

To work with Draft there are three prerequisites:

  1. Downloading the Draft CLI.
  2. Using Helm (The Kubernetes package manager) in your cluster.
  3. Setup a local Kubernetes cluster such as Minikube, Docker-for-desktop or MicroK8s.

The figure below depicts the Draft workflow, including the generation of the configuration files and the deployment lifecycle.

Deployments with Draft
Deployments with Draft

Draft comes with a set of packs that know what kind of dockerfiles and manifests are needed for all the popular programming languages such as Python, Java, Go etc. These packages are used automatically by Draft to generate a Dockerfile and a Helm chart when the draft create command is issued. The pack architecture for the supported languages is extensible, so developers can write their own extensions according to their team and application needs. Port forwarding to localhost is also provided by Draft. The resulting image deployed to the target cluster is sent via the Helm chart when issuing the command draft up.

When a change is happening to the source code of the application, the Draft daemon on the developer machine updates the Kubernetes cluster automatically by building a new container and then deploying again without any other developer action. There are multiple examples in different programming languages (e.g. Python, Go, Java) provided by Draft in the examples directory if you want to experiment using Draft with an existing application.

The main features of the Draft are:

  • The generated configuration files, such as Helm chart and Dockerfile are fully configured and can be customized further by hand to fully match the application developed. You can also develop your own custom language packs if you want to use Draft with an exotic programming language.
  • A wide range of supported languages are already included without the need to write dockerfiles or Kubernetes manifests. This can help developers who are just now starting their journey with Docker and Kubernetes.
  • All the build and deployment configuration is stored along the application source code directory making each application self-described.
  • It is very easy to modify/extend/adopt the configuration files into a full-blown CI/CD pipeline.
  • All builds are executed locally. No other external or cloud system is needed to work with Draft.
  • Draft automatically monitors local code changes and triggers cluster deployments. This feature is configurable and can be enabled/disabled by the developer.
  • Draft can also work with remote Kubernetes cintlusters if your local kubeconfig points to a remote cluster.

Draft was briefly mentioned in our AKS tutorial, but it was only used for the Helm chart generation. In the next tutorial, you will see the local workflow process as well.

Draft development tutorial

There are many options to install Draft either as a standalone binary or using a package depending on your development environment. In this tutorial, Draft installation will be completed using the standalone library for Linux. The installation steps are:

  1. Download the latest version of Draft
  2. Unpack it and place it to your $PATH
  3. Verify draft executable and version

Example installation:

wget https://azuredraft.blob.core.windows.net/draft/draft-v0.16.0-linux-amd64.tar.gz
tar -zxvf draft-v0.16.0-linux-amd64.tar.gz
sudo mv linux-amd64/draft /usr/local/bin/draft
draft version

After successful installation of Draft, you need to initialize it by executing the following command:

draft init

You need to run this command only once (the first time you install draft).

Application Configuration

After successful installation and configuration of Draft and its prerequisites, the next step is to deploy an application in your Kubernetes cluster. There are multiple examples in different programming languages (e.g. Python, Go, Java) provided by Draft in the examples directory. For this tutorial, instead of using existing samples, we have created a Java REST service based on Spring-Boot that returns a greeting message in JSON format. You can download the example project here.

If you look at the source code of the project you will notice that there are no Kubernetes manifests present or Helm charts. The application does not even include a Dockerfile.
In order to generate the needed files, you can use the command:

draft create

Draft will analyze the source code, notice that it is a Java application, and it will create for you:

  1. A Dockerfile,
  2. Kubernetes manifests,
  3. A Helm chart to package the manifests.

You can view (and edit) the generated files on your own:

pliakas@zouzou:~/Projects/Articles/draft-tutorial$ ls -laF
total 44
drwxr-xr-x 5 pliakas pliakas 4096 Μαρ   4 17:29 ./
drwxr-xr-x 4 pliakas pliakas 4096 Μαρ   3 21:27 ../
drwxr-xr-x 3 pliakas pliakas 4096 Μαρ   4 17:22 charts/
-rw-r--r-- 1 pliakas pliakas  304 Μαρ   4 17:25 Dockerfile
-rw-r--r-- 1 pliakas pliakas   50 Μαρ   4 17:22 .dockerignore
-rw-r--r-- 1 pliakas pliakas   25 Μαρ   4 17:22 .draftignore
-rw-r--r-- 1 pliakas pliakas    0 Μαρ   4 17:22 .draft-tasks.toml
-rw-r--r-- 1 pliakas pliakas  222 Μαρ   4 17:22 draft.toml
drwxr-xr-x 8 pliakas pliakas 4096 Μαρ   4 17:18 .git/
-rw-r--r-- 1 pliakas pliakas 1640 Μαρ   4 17:17 pom.xml
-rw-r--r-- 1 pliakas pliakas   94 Μαρ   3 21:27 README.md
drwxr-xr-x 4 pliakas pliakas 4096 Μαρ   4 17:16 src/

The charts/ and Dockerfile assets created by Draft default to a basic Java configuration. To align with the internalPort service value in charts/draft-tutorial/values.yaml, this Dockerfile exposes port 4567 from the container. The application is also using openjdk11 so modify the Dockerfile with your IDE or text editor as seen below:

FROM maven:3-jdk-11 as BUILD

COPY . /usr/src/app
RUN mvn --batch-mode -f /usr/src/app/pom.xml clean package

FROM openjdk:11-jre-slim
ENV PORT 4567
EXPOSE 4567
COPY --from=BUILD /usr/src/app/target /opt/target
WORKDIR /opt/target

CMD ["/bin/bash", "-c", "find -type f -name '*.jar' | xargs java -jar"]

The draft.toml file is a Draft specific file that provides the basic configuration about the application such as its name, the namespace it will be deployed to, and whether to deploy the application automatically when local files change.

[environments]
  [environments.development]
    name = "draft-tutorial"
    namespace = "default"
    wait = true
    watch = false
    watch-delay = 2
    auto-connect = false
    dockerfile = "Dockerfile"
    chart = ""

Detailed information about the structure of the draft.toml file can be found at the Draft reference documentation. A .draftignore file is created for elements we want to exclude tracking when watching for code changes.

Draft will also create a .dockerignore file to ensure the Docker context ignores files and directories that are not necessary. Feel free to edit this file further to match your application.

The .draft-tasks.toml file allows you to configure tasks to be run before draft up (pre-uptasks), after draft up (post-up tasks), or after draft delete (cleanup tasks). This file is empty by default. It can be used as a mini CI/CD pipeline that you can run locally when developing with Draft.

Application Deployment in a local Kubernetes cluster

After configuring the application, you can deploy it to your Kubernetes cluster with the command:

pliakas@zouzou:~/Projects/Articles/draft-tutorial$ draft up
Draft Up Started: 'draft-tutorial': 01D54PRB86WDFWZBGG94CEE3DA
draft-tutorial: Building Docker Image: SUCCESS ⚓  (136.0224s)
draft-tutorial: Releasing Application: SUCCESS ⚓  (2.4213s)
Inspect the logs with `draft logs 01D54PRB86WDFWZBGG94CEE3DA`

The above command will build all Docker images, then create the needed charts and deploy them to the Kubernetes cluster that is mentioned in your local kubeconfig.

You can verify the application deployment by using your favorite kubectl commands or any type of graphical dashboard that you might have.

kubectl get pods

You can verify that your application works by making an http request with curl/wget or by opening a browser and using the following endpoint.

curl http:///draft/api/v1.0/ 

Draft can also stream the logs from the container with the connect command to your standard output.

draft connect

Draft provides fast re-deployment of your application for any code modification that you make. After you edit a source code file, you can re-issue the command:

draft up 

Draft identifies that the Helm release exists and will perform a helm upgrade install of helm install, which speeds-up the redeployment procedure.

Finally, you can terminate and remove the deployed application from the Kubernetes cluster by issuing the command:

draft delete

In summary, Draft is a very handy solution for local Kubernetes development which is very easy to use even in an application that doesn’t have a Dockerfile.

Local Kubernetes development with Skaffold

Skaffold is a tool created by Google that helps with continuous development and deployment of Kubernetes applications specifically aimed at developers.

At the most basic level, Skaffold works similarly to Draft. It:

  1. Creates Kubernetes configuration files for your apps
  2. Deploys your application to a local or remote cluster
  3. Monitors your source code and automatically re-deploys when needed
  4. Steams logs from your deployed pods to your local terminal

But Skaffold also offers several advanced features on top of this basic functionality. First of all, Skaffold provides an extensible pluggable architecture, allowing developers to choose the appropriate tools for each step involved in building and deploying their application. Skaffold also attempts to provide portability for CI integrations among different build systems, image registries and deployment tools.

Skaffold can also interact with Docker registries by pushing your application image. Unlike Draft, Skaffold can work with plain Kubernetes manifests (using Helm as a package manager is optional).

The figure below depicts the Skaffold workflow, including the generation of the configuration files and the deployment lifecycle in both aforementioned execution modes.

Deployments with Skaffold
Deployments with Skaffold

Skaffold can be used in two different modes:

  1. Development mode by issuing the command “skaffold dev”. In development mode, it monitors the changes in source code, automatically generates all appropriate Docker images and deploys the images to the Kubernetes cluster. In addition, it provides logs streaming from deployed applications.
  2. Deployment mode by issuing the command skaffold run. In this mode, Skaffold runs a pipeline only once and exits on any errors in the pipeline. This is useful because it can be used as a sanity check after the completion of the development of the application.

The main features of the Scaffold are:

  • Monitors local changes in your source code and automatically triggers build/push deployments to local or remote Kubernetes clusters.
  • It supports remote and local Docker engines and registries. (Note that using remote Docker engines may increase build times significantly.)
  • Can track dependencies between applications and automatically deploy on what was changed.
  • Supports existing tooling and workflows with the ability to build and deploy APIs that make each implementation composable to support various different workflows.
  • Can be used from multiple different environments, such as the local developer environment or any CI since it is CI vendor independent (no need for any plugin for Kubernetes or other tools).
  • Has a pluggable architecture to allow for different implementations of the stages and build profiles to switch tools depending on the case.
  • Supports applications consisting of multiple components if you use Gradle or Maven.

One of the main advantages of Skaffold is the number of integrations it provides. It has native support for:

Buildkit
Bazel
Jib
Kaniko
Google cloud build
Helm
Kustomize

The skaffold.yml file, which is the main configuration file, has several options that you can use to change the way your application is built and deployed. It is essentially a mini CI/CD solution for local development that can also take care of testing or pushing images.

Skaffold development tutorial

In this tutorial, Skaffold installation will be completed using the standalone executable, for Linux. The installation steps are:

  1. Download the latest version of Skaffold
  2. Unpack it and place it to your $PATH
  3. Verify Skaffold executable and version

Example installation:

wget https://github.com/GoogleContainerTools/skaffold/releases/download/v0.24.0/skaffold-linux-amd64
chmod +x skaffold-linux-amd64
sudo mv skaffold-linux-amd64 /usr/local/bin/skaffold
skaffold version

Application Configuration

For this tutorial, a Spring-Boot application will be used. It provides one endpoint that returns a simple greeting message in JSON format. You can find the example application here.

In order to run the application, we first need to create a Dockerfile file to containerize our application. Here is an example:

FROM maven:3-jdk-11 as BUILD

COPY . /usr/src/app
RUN mvn --batch-mode -f /usr/src/app/pom.xml clean package

FROM openjdk:11-jre-slim
ENV PORT 42050
EXPOSE 42050
COPY --from=BUILD /usr/src/app/target /opt/target
WORKDIR /opt/target

CMD ["/bin/bash", "-c", "find -type f -name '*.jar' | xargs java -jar"]

In addition to the Dockerfile, we will also create a k8s-app.yml file in order to generate the Skaffold configuration file. A simple version of k8s-app.yml file is shown below:

apiVersion: v1
kind: Pod
metadata:
  name: skaffold-demo
spec:
  containers:
  - name: skaffold-demo
    image: roukou/skaffold-demo

Now, the Skaffold configuration files can be generated by issuing the command:

skaffold init

This command will generate a skaffold.yaml file containing all needed information in order to build and deploy the application to the desired Kubernetes cluster. The generated skaffold.yaml file is:

apiVersion: skaffold/v1beta6
kind: Config
build:
  artifacts:
  - image: roukou/skaffold-demo
deploy:
  kubectl:
    manifests:
    - k8s-app.yml

Application Deployment

After successful creation of the configuration files, there are two options to deploy the desired application. During development time, we can use the command:

skaffold dev --default-repo localhost:32000

The above command will continuously deploy your application when a file is changed (speeding up development). It monitors the source code files for every change and makes fast re-deployments.

If you want to deploy once and don’t track code changes, you need the run command:

skaffold run --default-repo localhost:32000

After a successful deployment you can check if your application has been deployed in your Kubernetes cluster using the familiar kubectl commands:

kubectl get pods

You can check that the application is running by using the command:

curl http:///skaffold/api/v1.0 

Alternatively, you can check your Kubernetes dashboard to ensure your application has been deployed as expected.

Finally, you can remove your deployed application with the delete command:

skaffold delete

In summary, Skaffold is a very flexible solution that does not require Helm and is very extensive on what kind of integrations it can be used with.

Local Kubernetes development with Garden

Garden is an open-source development tool aiming to provide a development environment to develop and test applications on Kubernetes (local or remote), or to any platform that supports serverless functions.

Garden allows developers to focus on the development cycle and provides an environment to build, deploy, and test applications faster, without spending time for configuration and deployment.

At first glance, Garden behaves in a similar manner to Draft and Skaffold. It allows you to deploy your source code to your local cluster and sync your deployments as you change the source code. It also supports extra deployment steps such as unit testing and can take advantage of Helm charts.

Garden introduces a new way to describe your application, called Stack Graph. It has six main organization units: projects, providers, modules, services, tasks and tests. A project is the root of the “graph” and consists of one or more modules that support multiple providers. Tasks can be used for dependencies of services. More details on stack graph can be found on a dedicated blog post by the Garden team.

With the stack graph, you essentially create a dependency tree between your application components. This means that when Garden deploys your application, it knows how all the parts fit together and can deploy only what was actually changed. You can think of Garden like a Kubernetes build system that behaves similarly to Make, Maven, Rake, Gulp, etc., but groups cluster services instead of source code.

The figure below depicts the Garden workflow, including the generation of the configuration files.

Development with Garden
Development with Garden

Garden combines build, test and deploy phases into a single command called garden dev. This provides a fast and convenient way to speed up the development feedback loop. This command will always build, test, and deploy modules in the right order so that all dependencies are respected. Furthermore, Garden is able to monitor the application source code files and automatically update any running services.

The main features of Garden are:

  • An integrated development and deployment framework for building, testing and deploying services efficiently.
  • Support of automatic cluster re-deployment for the application when source code is changing.
  • Support for multi-module and multi-services operations (tree of dependencies).
  • A graphical dashboard for dependencies.
  • Ability to run tasks (e.g. database migrations as part of the build flow).
  • Native support for Helm and OpenFass deployments
  • Hot-reload feature (mostly for dynamic programming languages as well as Go and Spring Boot) where source code is sent directly to running containers.
  • Streaming of container logs to your terminal.
  • File watching and hot reloading of code is also available for remote clusters.

Garden also has its own comparison against Skaffold. In summary, Garden aims to be a more generic solution (Kubernetes is just one deployment target, other targets such as Docker-swarm are planned).

Notice also that unlike Draft (Microsoft) and Skaffold (Google), Garden is a company with a single product and some of the planned features are expected to be enterprise/paid such as faster in-cluster building.

Garden development tutorial

In this tutorial, Garden installation will be completed using the standalone executable for Linux. The installation steps are:

  • Download the latest version of Garden
  • Unpack it and place it to your $PATH
  • Verify the executable and version

Example installation:

wget https://github.com/garden-io/garden/releases/download/v0.9.9/garden-v0.9.9-linux-amd64.tar.gz
tar -zxvf garden-v0.9.9-linux-amd64.tar.gz
cp -r linux-amd64/* ./local/bin/
garden -version

Application Configuration

For this tutorial, a spring-boot application will be used, which can be found here. It provides one endpoint that returns a simple greeting message in JSON format. A garden.yml file is needed in the parent directory of your project:

project:
  name: garden-tutorial
  environments:
    - name: local
      providers:
        - name: local-kubernetes

The local-kubernetes plugin automatically detects an existing Kubernetes installation and sets the appropriate context for connecting to the local Kubernetes instance. It uses the local-kubernetes plugin by default, but you can configure it explicitly in the project-level garden.yml as follows:

project:
  name: garden-tutorial
  environments:
    - name: local
      providers:
        - name: local-kubernetes
          context: minikube

In order to run the application, we first need to create a Dockerfile file to containerize our application, similar to the previous example for Skaffold.

FROM maven:3.6.0-jdk-11-slim
WORKDIR /app

COPY pom.xml /app/pom.xml
RUN mvn dependency:resolve

RUN mkdir -p /app/target
COPY src /app/src
# RUN mvn install

ENV JVM_OPTS "-XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1"
RUN mvn install
# RUN mvn compile
CMD ["mvn", "spring-boot:run"]

EXPOSE 42052

Furthermore, inside the directory of the project module/service an additional garden.yml file is needed. For each module in your project this configuration file is required and defines what type of module is supported (e.g. single container, Helm chart, Maven module etc). For the above module the garden.yml can be configured as follows:

module:
  description: Spring Boot to demonstrate garden deployment.
  type: container
  name: roukou
  hotReload:
    sync:
      - target: /app/target
        source: target
  services:
    - name: roukou
      ports:
        - name: http
          containerPort: 8080
          servicePort: 8080
      healthCheck:
        httpGet:
          path: /actuator/health
          port: http
      ingresses:
        - path: /garden/api/v1.0/
          port: http

A module may define one or more services. Services are deployed when running garden deploy or garden dev. Finally, you can verify that your configuration is correct by issuing the command:

garden get status

The output of the command must be similar to this:

providers:
  local-kubernetes:
    ready: false
    needUserInput: false
    dashboardPages: []
services:
  roukou:
    ingresses:
      - hostname: garden-tutorial.192.168.99.100.nip.io
        path: /
        port: 80
        protocol: http
    state: missing

Application Deployment

Garden support different workflows, depending on your requirements. During development, you can use the command:

garden dev

And the output will be:

Good afternoon! Let's get your environment wired up...

✔ roukou                    → Building roukou:v-c4d1c84056... → Done (took 0.4 sec)
✔ roukou                    → Deploying version v-c4d1c84056... → Done (took 67 sec)
    ℹ roukou                    → Service deployed
    → Ingress: http://garden-tutorial.192.168.99.100.nip.io/garden/api/v1.0
Garden dashboard and API server running on http://localhost:38365

Waiting for code changes

To verify that your application is deployed correctly to your local Kubernetes cluster, you can see it in the Garden dashboard by opening the link http://localhost:38365. Garden provides a graph showing all the dependencies of your applications, similar to the below screenshot:

Garden dashboard
Garden dashboard

You can check that your application is up and running using the command:

curl http://garden-tutorial.192.168.99.100.nip.io/garden/api/v1.0/

Alternatively, you can verify that the application was deployed into Kubernetes using the command:

kubectl get all --all-namespaces

Every time you are changing files, Garden is monitoring your files and re-builds & re-deploys the application. You can also build and deploy the application separately by issuing the commands garden build and garden deploy accordingly. Finally, Garden supports advanced features such as hot deployments without having to restart the application using the command:

garden deploy --hot=roukou

Conclusions

In this article, we have seen 3 tools for local Kubernetes development. They are used to deploy your source code to your local cluster and can continuously re-deploy when source code changes take place.

Draft is the best solution if you are just starting to learn about containers and Kubernetes clusters. It is the easiest to use if you have a plain application and want to auto-generate Dockefiles and Kubernetes manifests.

Draft advantages:

  • The most user-friendly
  • Very easy to setup and use
  • Language packs for most programming languages can be used to create dockerfiles and Helm charts
  • Minimal and lightweight
  • Created by Microsoft/Azure

Draft disadvantages:

  • Tied with the Helm package manager which is not used in all companies
  • Very simplistic. It just packages docker images and deploys them to a cluster. If you want any additional features you need to look to the other two options.

If you think that Draft is very basic, then Skaffold is much more flexible and can work as a mini CI/CD solution for local deployments. Note however that Skaffold has a strong flavor of Google (Bazel) and Java technologies (Jib/Gradle/Maven).

Skaffold advantages:

  • Flexible workflows with testers/deployers/push strategies
  • Can also tag and push Docker images with customizable tagging
  • Built-in integrations with several builders/deployers
  • Can deploy with Helm or Kustomize of plain kubectl
  • Support for build profiles
  • Extensible
  • Created by Google

Skaffold disadvantages:

  • Might be overkill for simple applications
  • Complex setup (compared to Draft)

Garden is the underdog and takes a much more interesting approach by offering more than local deployments in the form of an application graph that can track dependencies and re-deploy only what is changed.

Garden.io advantages:

  • Create a stack graph that explicitly describes your application dependencies
  • Support for testing and custom tasks allow for a flexible deployment model
  • Specific support for OpenFaas
  • Pushing and tagging of images

Garden.io disadvantages:

  • Company is a startup. Some advanced features are Paid/Enterprise
  • Might be overkill for simple applications
  • More complex setup compared to Draft/Skaffold; there is a learning curve for new concepts.

How do you deploy on Kubernetes as a developer to your local cluster? Let us know in the comments below.

How useful was this post?

Click on a star to rate it!

Average rating 5 / 5. Vote count: 1

No votes so far! Be the first to rate this post.

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments