Preview Environments
Deploy Pull Requests to cluster namespaces
In the previous guide we have seen how you can handle deployments to predefined environments (QA/Staging/production).
Another type of environments that you should manage is dynamic temporary environments for each pull request. For this type of environments it is best if you create dynamically an environment when a Pull Request is created and tear it down when the Pull Request is closed.
This way each developer is working in isolation and can test their feature on its own. This pattern comes in contrast with the traditional way of reusing static preexisting environments.
With Kubernetes you don’t need to book and release specific test environments any more. Testing environments should be handled in a transient way.
Preview environments with Kubernetes
There are many ways to create temporary environments with Kubernetes, but the simplest one is to use
different namespaces, one for each pull request. So a pull request with name fix-db-query
will
be deployed to a namespace called fix-db-query
, a pull request with name JIRA-1434
will be deployed to a namespace called
JIRA-1434
and so on.
The second aspect is exposing the environment URL so that developers and testers can actually preview the application deployment either manually or via automated tests.
The two major approaches here are with host-based URLs or path based URLs.
- In host based urls, the test environments are named
pr1.example.com
,pr2.example.com
and so on - with path based URLs, the test environments are named
example.com/pr1
,example.com/pr2
and so on
Both approaches have advantages and disadvantages. Path based URLs are easier to setup but may not work with all applications (since they change the web context). Host based URLs are more robust but need extra DNS configuration for the full effect.
In Kubernetes clusters, both ways can be setup via an Ingress.
The example application
The application we will use can be found at https://github.com/codefresh-contrib/unlimited-test-environments-source-code. It is a standard Java/Spring boot application with the following characteristics.
- It has integration tests that can be targeted at any host/port. We will use those tests as smoke test that will verify the preview environment after it is deployed
- It comes bundled in a Helm chart
- It has an ingress configuration ready for path based URLs
We are using the Ambassador gateway as an ingress for this example, but you can use any other compliant Kubernetes Ingress.
Here is the ingress manifest
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: "simple-java-app-ing"
annotations:
kubernetes.io/ingress.class: {{ .Values.ingress.class }}
spec:
rules:
- http:
paths:
- path: {{ .Values.ingress.path }}
backend:
serviceName: simple-service
servicePort: 80
The path of the application is configurable and can be set at deploy time.
Creating preview environments for each pull request
Each time a Pull Request is created we want to perform the following tasks:
- Compile the application and run unit tests
- Run security scans, quality checks and everything else we need to decided if the Pull request is valid
- Create a namespace with the same name as the pull request branch. Deploy the pull Request and expose it as a URL that has the same name as the branch as well
Here is an example pipeline that does all these tasks:
This pipeline has the following steps
- A clone step to fetch the source code of the application
- A freestyle step that runs Maven for compilation and unit tests
- A build step to create the docker image of the application
- A step that scans the source code for security issues with Snyk
- A step that scans the container image for security issues with trivy
- A step that runs integration tests by launching the app in a service container
- A step for Sonar analysis
- A step that clones a second Git repository that has the Helm chart of the app
- A step that deploys the source code to a new namespace.
- A step that adds a comment on the Pull Request with the URL of the temporary environment
- A step that runs smoke tests against the temporary test environment
Note that the integration tests and security scans are just examples of what you can do before the Pull Request is deployed. You can insert your own steps that check the contents of a Pull Request.
Here is the whole YAML definition
codefresh.yml
version: "1.0"
stages:
- "prepare"
- "verify"
- "deploy"
steps:
main_clone:
title: "Cloning repository"
type: "git-clone"
repo: "codefresh-contrib/unlimited-test-environments-source-code"
revision: "${{CF_REVISION}}"
stage: "prepare"
run_unit_tests:
title: Compile/Unit test
stage: prepare
image: 'maven:3.5.2-jdk-8-alpine'
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository package
build_app_image:
title: Building Docker Image
type: build
stage: prepare
image_name: kostiscodefresh/spring-actuator-sample-app
working_directory: ./
tag: '${{CF_BRANCH}}'
dockerfile: Dockerfile
scan_code:
title: Source security scan
stage: verify
image: 'snyk/snyk-cli:maven-3.6.3_java11'
commands:
- snyk monitor
scan_image:
title: Container security scan
stage: verify
image: 'aquasec/trivy'
commands:
- trivy image docker.io/kostiscodefresh/spring-actuator-sample-app:${{CF_BRANCH}}
run_integration_tests:
title: Integration tests
stage: verify
image: maven:3.5.2-jdk-8-alpine
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=http://my-spring-app -Dsonar.organization=kostis-codefresh-github
services:
composition:
my-spring-app:
image: '${{build_app_image}}'
ports:
- 8080
readiness:
timeoutSeconds: 30
periodSeconds: 15
image: byrnedo/alpine-curl
commands:
- "curl http://my-spring-app:8080/"
sonar_scan:
title: Sonar Scan
stage: verify
image: 'maven:3.8.1-jdk-11-slim'
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository sonar:sonar -Dsonar.login=${{SONAR_TOKEN}} -Dsonar.host.url=https://sonarcloud.io -Dsonar.organization=kostis-codefresh-github
clone:
title: "Cloning repository"
type: "git-clone"
repo: "codefresh-contrib/unlimited-test-environments-manifests"
revision: main
stage: "deploy"
deploy:
title: Deploying Helm Chart
type: helm
stage: deploy
working_directory: ./unlimited-test-environments-manifests
arguments:
action: install
chart_name: simple-java-app
release_name: my-spring-app
helm_version: 3.2.4
kube_context: myawscluster
namespace: ${{CF_BRANCH_TAG_NORMALIZED}}
cmd_ps: '--create-namespace --wait --timeout 5m'
custom_values:
- 'image_tag=${{CF_BRANCH_TAG_NORMALIZED}}'
- 'replicaCount=3'
- 'ingress_path=/${{CF_BRANCH_TAG_NORMALIZED}}/'
add_pr_comment:
title: Adding comment on PR
stage: deploy
type: kostis-codefresh/github-pr-comment
fail_fast: false
arguments:
PR_COMMENT_TEXT: "[CI] Staging environment is at https://kostis.sales-dev.codefresh.io/${{CF_BRANCH_TAG_NORMALIZED}}/"
GIT_PROVIDER_NAME: 'github-1'
run_smoke_tests:
title: Smoke tests
stage: deploy
image: maven:3.5.2-jdk-8-alpine
working_directory: "${{main_clone}}"
fail_fast: false
commands:
- mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=https://kostis.sales-dev.codefresh.io/${{CF_BRANCH_TAG_NORMALIZED}}/ -Dserver.port=443
The end result of the pipeline is a deployment on the path that has the same name as the pull request branch. For
example if my branch is named demo
then a demo namespace is created on the cluster and the application
is exposed on the /demo/
context:
The environment is also mentioned as a comment in the Pull Request UI in Github:
As explained it the previous guide for Pull Requests, we want to make this pipeline applicable only to PR open event and PR sync events (which capture commits that happen on an existing pull request).
Therefore you need to setup your triggers with the same checkboxes shown in the picture above.
Cleaning up temporary environments
Creating temporary environments is very convenient for developers but can be very costly for your infrastructure if you use a cloud provider for your cluster. For cost reasons and better resource utilization it is best if temporary environments are destroyed if they are no longer used.
While you can run a batch job, that automatically deletes old temporary environments, the optimal approach is to delete them as soon as the respective Pull Request is closed.
We can do that with a very simple pipeline that has only one step:
Here is the pipeline definition:
codefresh-close.yml
version: "1.0"
steps:
delete_app:
title: Delete app
type: helm
arguments:
action: auth
helm_version: 3.2.4
kube_context: myawscluster
namespace: ${{CF_BRANCH_TAG_NORMALIZED}}
commands:
- helm delete my-spring-app --namespace ${{CF_BRANCH_TAG_NORMALIZED}}
- kubectl delete namespace ${{CF_BRANCH_TAG_NORMALIZED}}
The pipeline just uninstalls the Helm release for that namespace and then deletes the namespace itself.
To have this pipeline run only when a Pull Request is closed here is how your trigger should look:
Notice that with this setup the pipeline will run when the pull request was closed regardless of whether it was merged or not (which is exactly what you want as in both cases the test environment is not needed anymore).
Seeing all environments in the Codefresh GUI
You can combine the pipeline above with any Codefresh GUI dashboard if you want to see all your temporary environments in a single view.
For more information see the documentation on dashboards such as the Environment dashboard, the Helm promotion dashboard and the GitOps dashboard