The Challenge : Microservice Integration Testing
The move to microservices brings new challenges with regards to testing our systems. In theory each microservice should be able to operate in isolation. But in practice a service usually isn’t worth much without its counterparts. On the other hand – bringing up the full system topology for testing just one service cancels out the benefits of modularity and encapsulation that microservices are expected to bring.
The challenge here is verifying that integration with other services isn’t broken. We want to do this as early as possible. And we don’t want to to replicate the complex production environment for this. Traditionally this verification was the function of integration or so-called end-to-end tests. But the fact is that the more complex our systems grow – the less value end-to-end tests bring. The sheer amount of interdependencies leads to multiple false negatives and excessively long execution times. Effectively making such tests too expensive to manage and debug.
There’s even an accepted concept of a test pyramid (first described by Mike Cohn in his book ‘Succeeding with Agile’) which basically says that for optimal effect you should have much more low-level unit tests than high-level end-to-end tests.
Psst, read this article and make sure to check out Codefresh, it’s the best CI for Docker out there.
Unit tests are great! But with all the benefits they bring – they provide no value for testing integration with other services.
So how do we allow each service team to iterate independently but still preserve the overall health of our system? How do we enable Continuous Delivery, small batch sizes and short feedback loops without breaking interdependent services on every other change?
One of the possible answers is Consumer-Driven Contract (CDC) testing. This testing strategy is based on a service evolution pattern first defined more than a decade ago. And it’s getting wider adoption now that distributed systems are becoming more and more commonplace.
Consumer-Driven Contracts:
I’ll try to explain this in very basic terms. Consumer-Driven Contracts are all about service-oriented relationship between… well, services. What this means is that instead of a provider dictating what the interface and service level will be (while consumers try their best to adapt) – now the consumers lead the dance. Each consumer defines what it expects the provider to deliver and the provider has to do the checking. It’s about shifting the responsibility for the integration to the providing side.
Which leads to the following flow:
In business terms this is usually described as ‘putting the customer first’ or ‘listening to your customers’. Because in order to provide great service we need to do what our customers expect and need. Not what we may assume the right thing to do is.
When talking about evolving microservices – this becomes especially important in a large enterprise setting where each service may be developed by a separate team. Sometimes these teams may also reside in different geographical and temporal regions. This prevents the immediacy of communication and makes collaborative evolution of business functionality ever more challenging.
Contract Testing Frameworks
Consumer-driven contracts can of course be managed by investing in communication and collaboration between teams. And also by using structured serialization formats such as protobuf, thrift or messagepack. But to manage a defined flow – it’s great to have a framework. Especially if it’s an open-source one.
And such frameworks have indeed emerged. The most prominent and actively developed ones are Pact and Spring Cloud Contract. The latter is only targeting the JVM-based projects. Pact, on the other hand was initially written for Ruby but has since been ported to quite a number of languages, including Java, Go, Python and Javascript. Which makes it a perfect choice for a complex, polyglot microservice system.
Today we will see how to use Pact in order to define and verify the contract between 2 services. A consumer service written in Python. And a provider service written in Go. The testing will occur as a part of our CI/CD process – right inside the Codefresh pipeline.
Pact
So, how does Pact work? Well, it all starts with the consumer.
The developer of the consumer service writes a test. The test defines an interaction with a provider. This includes a state the provider should be in, the body of the request and the expected response. Based on this definition Pact creates and spins up a provider stub against which the test executes. The output of this test is one or more json files that look something like this:
{
"consumer": {
"name": "billy"
},
"provider": {
"name": "bobby"
},
"interactions": [
{
"description": "My test",
"providerState": "User billy exists",
"request": {
"method": "POST",
"path": "/users/login",
"headers": {
"Content-Type": "application/json",
},
"body": {
"username":"billy",
"password":"issilly"
}
},
"response": {
"status": 200,
}
},
],
"metadata": {
"pactSpecification": {
"version": "2.0.0"
}
}
}
These are the contracts, the pacts. They are now passed on to the provider service. This can be done by committing them to a shared git repo, by uploading to a shared file storage or by using a special Pact Broker application.
Once the contract gets updated – the provider has to test if it still obeys it. It now runs its own verification tests against a real-life version of its service, using the shared pact file. If all the interactions go as expected and all the tests succeed – we’re good to go. If not – the developers of the provider should notify the developers of the consumer. Then, together they can analyze what led to contract breach.
Our example:
We will be testing the integration of 2 small services.
The provider is the same service we’ve used in our Jenkins plugin example. It’s called ‘bringon’, is written in Go and is a mongoDB-backed registry of software build information.
Our consumer will be a tiny python-based client that currently knows only how to fetch build information by build number from the bringon service.
As consumer comes first in CDC – we’ll start with it.
The whole code of the consumer currently consists of one client.py file that has 2 functions in it. We’re only concerned with the function called ‘build’ – as this is the functionality we will be testing.
import requests
…
def getbuild(host, port, buildnum):
"""Fetch a build by number ."""
uri = 'http://' + host + ':' + port + '/builds/' + str(buildnum)
return requests.get(uri).json()
In order to generate a pact for it – we created a test file build_test.py which looks like this:
import atexit
import unittest
import client
from pact import Consumer, Provider
pact = Consumer('buildReader').has_pact_with(Provider('bringon'))
pact.start_service()
atexit.register(pact.stop_service)
class GetBuildInfoContract(unittest.TestCase):
def test_get_build(self):
true = True
expected = {
u'name':u'#3455',
u'completed': true, #boolean
u'info':{
u'coverage':30,
u'apiversion':0.1,
u'swaggerlink':u'http://swagger',
u'buildtime':230}
}
(pact
.given('build 3455 exists')
.upon_receiving('a request for build 3455')
.with_request('get', '/builds/3455')
.will_respond_with(200, body=expected))
with pact:
result = client.build(3455)
self.assertEqual(result, expected)
It’s pretty straightforward – we’re spinning up a mock service, defining an expected http response code and body, and then calling the client.build() routine to make sure the interaction occurs as expected.
If all goes well – a pact file named buildreader-bringon.json gets written in our working directory.
Now we can share this file with the bringon developers so they can test the pact against their service.
This can be done with the help of pact-go – the Golang port of the framework. The test would look something like this:
func TestPact(t *testing.T) {
go startInstrumentedBringon()
pact := createPact()
// Verify the Provider with local Pact Files
log.Println("Start verify ", []string{filepath.ToSlash(fmt.Sprintf("%s/buildreader-bringon.json", pactDir))},
fmt.Sprintf("http://localhost:%d/setup", port), fmt.Sprintf("http://localhost:%d", port), fmt.Sprintf("%s/buildReader-bringon.json", pactDir))
err := pact.VerifyProvider(types.VerifyRequest{
ProviderBaseURL: fmt.Sprintf("http://localhost:%d", port),
PactURLs: []string{filepath.ToSlash(fmt.Sprintf("%s/buildReader-bringon.json", pactDir))},
ProviderStatesSetupURL: fmt.Sprintf("http://localhost:%d/setup", port),
})
if err != nil {
t.Fatal("Error:", err)
}
}
Note this requires quite a bit more work. We need to implement the startInstrumentedBringon() function which starts our service with an additional ‘/setup’ endpoint for defining the service state. In our case this will be used for creating a build entry which fits our consumer’s expectations. And we’ll also need to create a Pact client object that takes care of verifying all the interactions. Like this:
func createPact() dsl.Pact {
// Create Pact connecting to local Daemon
log.Println("Creating pact")
return dsl.Pact{
Consumer: "buildreader",
Provider: "bringon",
LogDir: logDir,
PactDir: pactDir,
}
}
One downside of using pact-go in its current form is it requires you to run a daemon in the background. The daemon takes care of service initiation, shutdown and pact verification.
Not really a good fit for running one-off ephemeral processes inside containers.
So if all we want to do is test the pacts against our service – we can instead use the lightweight pact-provider-verifier utility that comes packed with pact-go.
Like this:
pact-provider-verifier --pact-urls <path_to>/buildreader-bringon.json --provider-base-url http://localhost:8091 --provider-states-setup-url http://localhost:8091/setup
Please note that in this case we’ll have to implement and build the ‘/setup’ endpoint as a part of our service. Which can be a good idea if we want to make our service ultimately testable.
The code for both services can be found on our Github:
bringon (or The Provider) :
https://github.com/codefreshdemo/bringon
buildreader (or The Consumer):
https://github.com/antweiss/cdc-pact-demo
Pact sources and examples are here:
https://github.com/pact-foundation
In the next installment of this blog we will show how to run the contract test as a part of your Codefresh pipeline.
And what’s your favorite strategy for testing microservice integration?
New to Codefresh? Schedule a FREE onboarding and start making pipelines fast. and start building, testing and deploying Docker images faster than ever.