Previewing dynamic environments
Deploy pull requests to cluster namespaces
In addition to deploying to predefined environments, for each pull request (PR), you may also need to deploy to dynamic environments, which are temporary, testing environments. For these types of environments, it is best to dynamically create an environment when a PR is created, and tear it down when the same PR is closed.
Each developer works in isolation to test their features. This pattern contrasts with the traditional way of reusing static preexisting environments.
With Kubernetes you don’t need to book and release specific test environments any more. Testing environments should be handled in a transient way.
Preview environments with Kubernetes
There are many options to create temporary environments with Kubernetes.
Namespaces for each PR
The simplest option is to use different namespaces, one for each PR. So, a PR with name
fix-db-queryis deployed to a namespace called
fix-db-query, and a PR with name
JIRA-1434, is deployed to a namespace called
JIRA-1434and so on.
Expose the environment URL
The second option is to expose the environment URL so that developers and testers can actually preview the application deployment either manually or via automated tests.
The two major approaches here are with host-based and path-based URLs:
- For host-based URLs, the test environments are named
pr2.example.comand so on
- For path-based URLs, the test environments are named
example.com/pr2and so on
Both approaches have advantages and disadvantages. Path-based URLs are easier to set up, but may not work with all applications, as they change the web context. Host-based URLs are more robust but need extra DNS configuration for the full effect.
In Kubernetes clusters, you can set up types of URLs via an Ingress.
- For host-based URLs, the test environments are named
You can find the application we will use at https://github.com/codefresh-contrib/unlimited-test-environments-source-code.
It is a standard Java/Spring boot application, that includes the following characteristics:
- It has integration tests that can be targeted at any host/port. We will use those tests as smoke test that will verify the preview environment after it is deployed
- It comes bundled in a Helm chart
- It has an ingress configuration ready for path-based URLs
We are using the Ambassador gateway as an ingress for this example, but you can use any Kubernetes-compliant ingress.
Here is the ingress manifest.
The path of the application is configurable and can be set at deploy time.
Creating preview environments for each PR
Each time a PR is created, we want to perform the following tasks:
- Compile the application and run unit tests.
- Run security scans, quality checks, and everything else we need to decide if the PR is valid.
- Create a namespace with the same name as the PR branch. Deploy the PR and expose it as a URL that has the same name as the branch.
Here is an example pipeline that does all these tasks:
This pipeline has the following steps:
- A clone step to fetch the source code of the application.
- A freestyle step that runs Maven for compilation and unit tests.
- A build step to create the Docker image of the application.
- A step that scans the source code for security issues with Snyk.
- A step that scans the container image for security issues with trivy.
- A step that runs integration tests by launching the app in a service container.
- A step for Sonar analysis.
- A step that clones a second Git repository with the Helm chart of the application.
- A step that deploys the source code to a new namespace.
- A step that adds a comment on the PR with the URL of the temporary environment.
- A step that runs smoke tests against the temporary test environment.
Note that the integration tests and security scans are just examples of what you can do before the PR is deployed. You can insert your own steps that check the content of a PR.
Here is the complete YAML definition:
The end result of the pipeline is a deployment to the path that has the same name as the PR branch. For
example, if my branch is named
demo, then a
demo namespace is created on the cluster and the application
is exposed on the
The environment is also mentioned as a comment in the PR UI in GitHub:
As explained in Pull Requests and branches guide, we want to make this pipeline applicable only to a PR-open event and PR-sync events that capture commits on an existing pull request.
Therefore, you need to set up your Git triggers with the same options selected as shown in the picture above.
Cleaning up temporary environments
Creating temporary environments is very convenient for developers, but can be very costly for your infrastructure if you use a cloud provider for your cluster. For cost reasons and better resource utilization, it is best to destroy temporary environments that are no longer used.
While you can run a batch job that automatically deletes old temporary environments, the optimal approach is to delete them as soon as the respective PR is closed.
We can do that with a very simple pipeline that has only one step:
Here is the pipeline definition:
The pipeline just uninstalls the Helm release for that namespace, and then deletes the namespace itself.
To have this pipeline run only when a PR is closed, here are the triggers to select:
With this setup, the pipeline runs when the PR is closed, regardless of whether it was merged or not (which is exactly what you want as in both cases the test environment is not needed anymore).
Viewing all environments in the Codefresh UI
You can combine the pipeline above with any Codefresh UI dashboard if you want to see all your temporary environments in a single view.
For more information, see:
How Codefresh pipelines work
Codefresh YAML for pipeline definitions
Steps in pipelines
Docker registry integrations for pipelines