This is an attempt to create the simplest demo to describe Consumer-Driven Contract Testing with Pact.
Pact introduces a (new) way to test the interactions (over the network) between different components without the need to run these components together within the same (shared) environment! In other words, it's an alternative to end-to-end testing, which is called consumer-driven contract testing.
In addition to that, it unambiguously addresses the problematic of test data. I do think that (traditional) tools that are supposed to help performing end-to-end testing, such as service virtualization, leave the question of test data - or more precisely the question of how to set the right test data at the right moment in order to correctly perform the testing of a specific interaction - as "the elephant in the room".
Pact addresses the problematic of test data with the key notion of Provider State. I do see this notion of Provider State as the cornerstone of Pact! But, first, with Pact, you have to understand that one (and only one) interaction - a single request-response pair - is tested at a time, independently. You never (ever) test any sequences of interactions! That means that for each interaction you have to define precisely in which state the provider is, and that's what is called the Provider State.
[To be continued...]
This scenario has been developed on Windows.
This scenario is using node, curl and jq.
This scenario is also using pact-stub-server and pact_verifier_cli.
node --version
v16.13.2
curl --version
curl 7.79.1 (Windows) libcurl/7.79.1 Schannel
Release-Date: 2021-09-22
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets
jq --version
jq-1.6
pact-stub-server --version
pact-stub-server v0.4.4
pact stub server version : v0.4.4
pact specification version: v3.0.0
pact_verifier_cli --version
pact_verifier_cli 0.9.7
pact verifier version : v0.9.7
pact specification : v4.0
models version : v0.2.7
1. The consumer gets (typically via a developer portal) the OpenAPI (f.k.a. Swagger) document of an API. For this demo, we will use the "Thingies API", thingies-api.oas2.yaml.
2. The consumer wants to use the "Thingies API" with her/his own test data. For instance when she/he sends the request:
GET localhost:8000/thingies/123
she/he expects to receive the following response:
{
"id": "123",
"name": "stuff"
}3. Therefore, the consumer creates a Pact file that follows her/his scenario/interaction, consumer.pact.json.
Warning. There is, of course, a risk that the consumer does not respect the actual behavior of the API when defining her/his scenario/interaction, that's why there is a verification step afterwards.
4. And then, the consumer can start a Pact Stub Server using the pact-stub-server command with the Pact file:
pact-stub-server --file consumer.pact.json --port 8000
in order to run her/his test script, e.g.consumer.test.cmd:
consumer.test.cmd
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 27 100 27 0 0 27 0 0:00:01 --:--:-- 0:00:01 114
"name": "stuff"
Passed.
Remark. At this stage, this test will obviously work as it is the same consumer that decides, within the test script, what the response should be, but also decides, within the Pact file run by the Pact Stub Server, what the response will be. This test is like a self-fulfilling prophecy! Again, that's why there is a verification step afterwards, during which the provider will verify that the Pact file created by the consumer makes sense or not.
5. Now, the consumer passes her/his Pact file, consumer.pact.json, to the provider so she/he can verify it.
6. The provider runs her/his implementation of the "Thingies API", e.g. provider.app.v1.js:
node provider.app.v1
Provider service is running at localhost:3000...
7. The provider can then verify the consumer Pact file, consumer.pact.json, with the Pact Verifier CLI using the pact_verifier_cli command. This tool will read the Pact file the other way around: it will send the request of the interaction to the actual implementation of the "Thingies API" and check if the actual response corresponds to the response defined by the consumer in the Pact file.
pact_verifier_cli --file consumer.pact.json --hostname localhost --port 3000
Given has one thingy with '123' as an thingyId
WARNING: State Change ignored as there is no state change URL provided
09:15:55 [WARN]
Please note:
We are tracking events anonymously to gather important usage statistics like Pact version and operating system. To disable tracking, set the 'PACT_DO_NOT_TRACK' environment variable to 'true'.
Verifying a pact between Thingies Consumer Example and Thingies Provider Example
get one thingy
returns a response which
has status code 200 (FAILED)
includes headers
"Content-Type" with value "application/json; charset=utf-8" (FAILED)
has a matching body (FAILED)
Failures:
1) Verifying a pact between Thingies Consumer Example and Thingies Provider Example Given has one thingy with '123' as an thingyId - get one thingy
1.1) has a matching body
/ -> Expected body Present(27 bytes) but was empty
1.2) has status code 200
expected 200 but was 404
1.3) includes header 'Content-Type' with value '"application/json; charset=utf-8"'
Expected header 'Content-Type' to have value '"application/json; charset=utf-8"' but was ''
There were 1 pact failures
The verification obviously fails as the test data invented by the consumer does not match the test/real data used by the provider.
This is where the Provider States come to the rescue by "allowing you to set up data on the provider by injecting it straight into the data source before the interaction is run, so that it can make a response that matches what the consumer expects." But, how does this "injection" work? No magic, just before sending the request of an interaction the pact_verifier_cli sends the Provider State, i.e. the free text representing the Provider State, to the provider using a POST / endpoint with the following JSON structure:
{
"action": "setup",
"params": {},
"state": "has one thingy with '123' as an thingyId"
}So, it means that, as a provider you have to manually map (by writing some new specific code) this long string representing the Provider State invented by the consumer with the injection of some specific test data. Something like this, implemented in provider.app.v2.js:
app.post('/', (req, res) => {
const providerState = req.body.state
switch(providerState) {
case "has one thingy with '123' as an thingyId":
thingies.push({
id: "123",
name: "stuff"
})
break
default:
res.status(404).end()
return
}
res.status(201).end()
})Warning. This is not production-ready code ;-)
So, you can see that this technique can be error-prone and maybe difficult to scale when a provider has a lot of customers, but this effort needed must be put in perspective with the work needed to maintain and prepare "connected test environment(s)" in order to perform valuable end-to-end testing. As a reminder, the goal of Consumer-Driven Contract Testing with Pact is to remove the need of end-to-end testing!
Running the updated version of implementation of the "Thingies API", provider.app.v2.js:
node provider.app.v2
We can re-run the pact_verifier_cli command specifying the URL of the POST endpoint using the --state-change-url option:
pact_verifier_cli --file consumer.pact.json --hostname localhost --port 3000 --state-change-url http://localhost:3000
Given has one thingy with '123' as an thingyId
09:18:15 [WARN]
Please note:
We are tracking events anonymously to gather important usage statistics like Pact version and operating system. To disable tracking, set the 'PACT_DO_NOT_TRACK' environment variable to 'true'.
Verifying a pact between Thingies Consumer Example and Thingies Provider Example
get one thingy
returns a response which
has status code 200 (OK)
includes headers
"Content-Type" with value "application/json; charset=utf-8" (OK)
has a matching body (OK)
Yeah!
- Skurrie, B. (2014, December 5). Enter the Pact Matrix. Or, how to decouple the release cycles of your microservices. REA Group Blog. https://www.rea-group.com/about-us/news-and-insights/blog/enter-the-pact-matrix-or-how-to-decouple-the-release-cycles-of-your-microservices/