Selenium Deployment Verification Testing of Docker’s Example Voting App

Join Our Newsletter!
Technical Guides

Selenium Deployment Verification Testing of Docker’s Example Voting App

8 min read
Dustin Van Buskirk

In this blog post, I will be focusing on running Deployment Verification Tests against Docker’s Example Voting App using Selenium and PyTest against the Kubernetes Helm Release we worked on in Part 1.

New to Codefresh? Get started with Codefresh by signing up for an account today!

This blog is Part 2 of a series The Perfect Pipeline.

Adding 4 New Build Steps:

  • GetKubernetesServicesEndpoints: We’re gathering the Endpoint IPs for vote and result service’s from Kubernetes
  • BuildingTestDockerImage: Building a Test Docker image with Python, PyTest and Selenium..
    PostDeploymentVerificationTests: Running a composition with Selenium Hub, Chrome Browser, Firefox Browser and Test image. This composition will utilize the open source Selenium Docker images to run our PyTest Deployment Verification Tests.
  • ArchiveDVTs: We’re generating HTML reports in the prior step and in this step archiving them in S3 and annotating our Test Docker image’s metadata with the URLs to the reports.

The diagram below represents the workflow discussed in this blog post.

Update to Existing YAML

Some updates have occurred to names and titles of the build steps but below are the critical updates.

Edit value of RELEASE_NAME for the HelmUpgrade build step to ${{HELM_RELEASE_NAME}}.
The Helm Release Name was templated for later use in new build steps.

Please be aware that WordPress may have modified symbols used the scripting below.

All examples below are available in this Github repository.

https://github.com/codefresh-io/example-voting-app

Create PyTest + Selenium python script

Create /tests/selenium/test_app.py example below..

import os
import time

import pytest

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.common.exceptions import (
    InvalidSelectorException,
    NoSuchElementException,
    ElementNotVisibleException,
    ElementNotInteractableException,
    WebDriverException)

vote_endpoint_ip = os.getenv('VOTE_ENDPOINT_IP')
result_endpoint_ip = os.getenv('RESULT_ENDPOINT_IP')

# Give Selenium Hub time to start
time.sleep(15)  # TODO: figure how to do this better

@pytest.fixture(scope='module')
def browser():
    browser_name = ip = os.getenv('BROWSER')
    browser = webdriver.Remote(
        command_executor='http://selenium_hub:4444/wd/hub',
        desired_capabilities={'browserName': 'chrome'},
    )
    yield browser
    browser.quit()


def test_confirm_vote_title(browser):
    browser.get("http://{}:80".format(vote_endpoint_ip))
    assert "Cats vs Dogs!" in browser.title


def test_confirm_vote_choice_form(browser):
    browser.get("http://{}:80".format(vote_endpoint_ip))
    element = browser.find_element(By.ID, 'choice')
    assert element.get_attribute('id') == 'choice'


def test_confirm_vote_button_a(browser):
    browser.get("http://{}:80".format(vote_endpoint_ip))
    element = browser.find_element(By.ID, 'a')
    assert element.get_attribute('id') == 'a'


def test_confirm_vote_button_b(browser):
    browser.get("http://{}:80".format(vote_endpoint_ip))
    element = browser.find_element(By.ID, 'b')
    assert element.get_attribute('id') == 'b'


def test_vote_click(browser):
    browser.get("http://{}:80".format(vote_endpoint_ip))
    browser.find_element(By.ID, 'a').click()
    WebDriverWait(browser, 3)


def test_confirm_result_title(browser):
    browser.get("http://{}:80".format(result_endpoint_ip))
    assert "Cats vs Dogs -- Result" in browser.title


def test_confirm_result(browser):
    browser.get("http://{}:80".format(result_endpoint_ip))
    element = browser.find_element(By.ID, 'result')
    assert element.get_attribute('id') == 'result'


def test_confirm_result_vote_tally(browser):
    browser.get("http://{}:80".format(result_endpoint_ip))
    element = browser.find_element(By.ID, 'result')
    assert 'No votes yet' not in element.text

There are simple tests for both the vote and result service’s webpage.

Using PyTest + Selenium Hub and allowing for multiple browsers.

For vote service we’re checking:

  • the value of the web page’s title is Cats vs Dogs!
  • the existence of the choice element
  • the existence of button a
  • the existence of button b

For result service we’re checking:

  • the value of the web page’s title is Cats vs Dogs -- Result
  • the existence of the result element

We’re also doing a simple button click() test on the vote web page and confirming the result web page tallied that vote.

  • execute click() of button a on vote web page
  • confirm result element of the result web page does not equal No votes yet as this would signify we didn’t tally the click() to button a on the vote web page.

Create Dockerfile for Test Docker image

Now we’ll create a new Dockerfile in the root folder of the repository

FROM python:3.4-alpine

RUN apk update && 
    apk add bash && 
    pip install selenium pytest pytest-html

We’re skipping copying in our tests in this Dockerfile and instead will use tests mounted from the GIT repository in our composition candidate or our composition build step.

Now that we have a Docker image ready for our Codefresh Pipeline we need to add a new build step to our Pipeline’s codefresh-matrix-pipeline.yml to build the Docker image.

Add Docker build Build Step for Test image

  BuildingTestDockerImage:
    title: Building Test Docker Image
    type: build
    image_name: codefresh/example-voting-app-tests
    working_directory: ./
    dockerfile: Dockerfile
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'

Add Get Kubernetes Service IP Build Step

Add GetKubernetesServicesEndpoints build step to get Endpoint IPs for vote and result service.

  GetKubernetesServicesEndpoints:
    title: Getting Kubernetes Services Endpoints
    image: codefresh/cfstep-helm:2.8.0
    commands:
      - bash -c 'IFS=" " read -a services <<< "${{SERVICES}}" && for service in "${services[@]}"; do external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc ${{HELM_RELEASE_NAME}}-${service} --namespace ${{KUBE_NAMESPACE}} --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export ${service^^}_ENDPOINT_IP=$external_ip; done'
  BuildingTestDockerImage:
    title: Building Test Docker Image
    type: build
    image_name: codefresh/example-voting-app-tests
    working_directory: ./
    dockerfile: Dockerfile
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'

To support the changes made in the matrix YAML file we need to add the following Environment Variables.

Add Helm Release Name and Kubernetes Services as Environment variables on the matrix pipeline.

  • HELM_RELEASE_NAME = example-voting-app <- YAML supports any name you’d like to use.
  • services = vote result <- space delimited services we would like to return Kubernetes External IPs for.

You will need to add additional browsers to the composition build step PostDeploymentVerificationTests if you’d like to support additional browsers.

Add Selenium Deployment Verification Build Step

Now we’ll add the composition build step which will stand up Selenium Hub, Chrome and Firefox browsers as supporting infrastructure and join in our Test Docker image as a composition candidate.

We’ve setup the composition candidate with a volume mount of our Github clone and passed in our Kubernetes Service’s IPs and browsers to test against.

When the tests have finished we will annotate the Test Docker image’s metadata with pass/fail.

  PostDeploymentVerificationTests:
    title: Running Selenium DVTs
    type: composition
    composition:
      version: '2'
      services:
        selenium_hub:
          image: selenium/hub
          ports:
            - 4444
          environment:
            - SE_OPTS=-debug
            - GRID_MAX_SESSION=5
        chrome_node:
          image: selenium/node-chrome
          ports:
            - 5900
            - 5555
          command: bash -c "sleep 5 && /opt/bin/entry_point.sh"
          depends_on: 
            - selenium_hub
          environment:
            - HUB_HOST=selenium_hub
            - REMOTE_HOST=http://chrome_node:5555
            - NODE_MAX_SESSION=5
            - NODE_MAX_INSTANCES=5
        firefox_node:
          image: selenium/node-firefox
          ports:
            - 5900
            - 5555
          command: bash -c "sleep 5 && /opt/bin/entry_point.sh"
          depends_on: 
            - selenium_hub
          environment:
            - HUB_HOST=selenium_hub
            - REMOTE_HOST=http://firefox_node:5555
            - NODE_MAX_SESSION=5
            - NODE_MAX_INSTANCES=5
    composition_candidates:
      test:
        image: ${{BuildingTestDockerImage}}
        working_dir: ${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}
        environment:
          VOTE_ENDPOINT_IP: ${{VOTE_ENDPOINT_IP}}
          RESULT_ENDPOINT_IP: ${{RESULT_ENDPOINT_IP}}
        command: bash -c 'IFS=" " read -a browserarray <<< "${{BROWSERS}}" && for browser in "${browserarray[@]}"; do BROWSER=$browser python -m pytest -vvv --html=./selenium-report-${browser}.html --self-contained-html ./tests/selenium/test_app.py; done'
        volumes:
          - '${{CF_VOLUME_NAME}}:/codefresh/volume'
    add_flow_volume_to_composition: true
    on_success:
      metadata:
        set:
          - '${{BuildingTestDockerImage.imageId}}':
              - SELENIUM_DVTS: true
    on_fail:
      metadata:
        set:
          - '${{BuildingTestDockerImage.imageId}}':
              - SELENIUM_DVTS: false

Add supported browsers as Environment variables on the matrix pipeline.

  • browsers = chrome firefox <- space delimited browsers you’d like to execute Selenium DVTS against.

You will need to add additional browsers to the composition build step PostDeploymentVerificationTests if you’d like to support additional browsers.

Add a S3 Bucket and Upload PyTest HTML Reports to S3 bucket

We generated HTML reports for each browser and will upload these to S3 bucket for the build and add the URLs to the Test Docker image’s metadata.

NOTICE

LINE 123: I’ve added an argument to make the uploads public --acl public-read

You may want to remove this argument if you use this build step in a private setting.

Prerequisites:

S3 bucket:
https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html

AWS Access Key and Secret Access Key:
https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys

Add YAML below to codefresh-matrix-pipeline.yml

  ArchiveSeleniumDVTReports:
    title: Archiving Selenium DVT Reports
    image: mesosphere/aws-cli
    working_directory: ./
    commands:
      - apk update
      - apk upgrade
      - apk add bash
      - bash -c 'IFS=" " read -a browserarray <<< "${{BROWSERS}}" && for browser in "${browserarray[@]}"; do BROWSER=$browser aws s3 cp ./selenium-report-${browser}.html s3://${{S3_BUCKETNAME}}/${{CF_BUILD_ID}}/selenium-report-${browser}.html --acl public-read; done'
    on_success:
     metadata:
        set:
          # I manually setup metadata for each browser
          - ${{BuildingTestDockerImage.imageId}}:
              - CHROME_SELENIUM_DVTS: "https://s3.${{AWS_DEFAULT_REGION}}.amazonaws.com/${{S3_BUCKETNAME}}/${{CF_BUILD_ID}}/selenium-report-chrome.html"
              - FIREFOX_SELENIUM_DVTS: "https://s3.${{AWS_DEFAULT_REGION}}.amazonaws.com/${{S3_BUCKETNAME}}/${{CF_BUILD_ID}}/selenium-report-firefox.html"

To support the changes made in the matrix YAML file we need to add the following Environment Variables.

Add AWS Region and Credentials to AWS_CLI Shared Secret

Account Settings -> Shared Configuration -> ADD CONFIGURATION CONTEXT -> Shared Secret

AWS_DEFAULT_REGION = aws-region-s3-bucket <- S3 bucket’s AWS Region.
AWS_ACCESS_KEY_ID = aws-access-key <- Access Key with read/write access to S3 bucket.
AWS_SECRET_ACCESS_KEY = aws-secret-access-key <- Secret Access Key for Access Key above.

Add S3 Bucket Name Global Variable

Codefresh Repository -> General (tab) -> Global Variables

S3_BUCKETNAME = example-voting-app <- named after my repository

We’ll be using this in future blog posts across the different pipelines for the Codefresh Repository.

Resources:

https://github.com/codefresh-io/example-voting-app/tree/add-selenium-tests#part-2-add-selenium-deployment-verification-testing-of-deployment

http://pytest-selenium.readthedocs.io/en/latest/index.html

https://github.com/SeleniumHQ/docker-selenium

https://github.com/SeleniumHQ/selenium/tree/master/py/test/selenium/webdriver/common

Planning for next blog to focus on Helm Chart Management using Codefresh’s embedded Chart Museum and use of Helm Repository in CI/CD workflow.

Codefresh Helm Repository

Hope, you find this tutorial useful. I look forward to your comments and any questions you have.

New to Codefresh? create a free Codefresh account and start building, testing and deploying Docker images faster than ever.

Ready to Get Started?
  • safer deployments
  • More frequent deployments
  • resilient deployments
Sign Up Book A Demo