Do Not Write Contract Tests
APIs are a contract, and one of the hardest parts of building an API is making sure that what you build follows that contract...perfectly.
For teams that go spec first, this means making sure the API they deploy matches that spec. For teams that write their code first, this means making sure they don't accidentally change the contract.
For most teams, meeting these challenges means testing, specifically contract testing.
Why API Testing is Hard
Testing APIs is traditionally done by running requests against the API server and making assertions about the responses.
Ideally, these requests:
- Fully exercise the surface area of the API (every endpoint is tested, with examples that yield every possible status code).
- Can replayed over and over again. This usually requires a separate testing environment which can be quite expensive.
- Do not include sensitive data
In practice, it is quite difficult to manually curate a suite of test requests that meet these requirements. It takes a ton of work to figure out which requests to run and to write good assertions about those responses. And, since APIs exist on the boundaries of systems, each endpoint may call out to other services in the infrastructure, speak with third-party APIs, or emit side effects like database writes that may need to be reversed later.
All that just to test a contract.
Live Contract Testing
Many teams are starting to invest in live contract testing. With live testing, there's no need to manually curate test cases. Instead, teams collect samples from their development and staging environments and run OpenAPI validators on the requests/responses their APIs handle. Whenever an endpoint behaves in an expected way, the request, response and other information needed to debug it are automatically logged.
This has several distinct advantages:
- Test data is 'real' and more representative of production traffic
- Within minutes of being deployed, the API handles enough traffic to provide confidence that the API works as expected.
- One time investment. Once you setup the environment to log discrepancies between the contract and real behavior, you get contract testing for free.
- No separate test environment or additional resources (mock databases + mock integrations) are needed.
If your APIs handle a reasonable amount of traffic while in development/staging live contract testing makes a lot of sense.
The downside of course is that you move your testing from build time to pre-deploy, but if you have good controls around CI and multiple environments that's less of a concern.
Coverage: The Elephant in the Room
Ok — I'm bought into the idea of live testing, but how do I know everything that matters was tested?
The major challenge in live testing are false positives. Just because you haven't seen any discrepancies between the API's behavior and its specification doesn't mean the API has been fully tested.
- Were there 1000 requests to endpoint A, B and C, but none to endpoint D?
- Was every possible status code observed for each endpoint?
- Was every instance of polymorphism in the spec been observed?
Without API coverage, which is like code coverage but for your API spec; live testing lacks statistical power. A lot of toolmakers are starting to think about API Coverage. Kyle Fuller of Dredd/Apiary spoke about contract testing, Dredd and API Coverage at ASC in Vancouver in October.
Live Testing + API Coverage
With API Coverage, live testing goes from being a cool idea to a practical and pragmatic replacement for manual contract testing.
- Allows teams to focus their manually written tests towards less exercised parts of the spec. Instead of writing 150 tests that cover 85% of the spec, write 10 that cover the 15% that is less exercised in development/staging.
- Gives teams the confidence that their API conforms to the API spec in all cases. Doing this manually would be very costly to write and maintain.
- Frees developers from writing contract tests and lets them focus on testing the functional / business requirements
API Coverage in Optic
In Optic, API coverage is measured by our development proxy. As traffic passes through the proxy, Optic heat-maps the traffic against the specification. If you have access to the developer preview channel you can try it in one of the latest builds by starting your API with the --coverage flag.
There are a few challenges right now that we would love your input on as we work through them:
- What is the proper verbiage and tone for the coverage report?
- There are many different categories of coverage. The simplest being coverage of all API paths. One level beyond that is coverage of all operations, then all possible responses. And finally you could provide a coverage report for polymorphism — is all the specified instances polymorphism observed at least once?
- Since we're watching the API traffic, it's possible to group usage and coverage by client using OpenTracing. This could be very valuable for regressions and determining which clients might be affected by certain classes of breaking changes.
If you're interesting in helping out, getting access to a preview build, or sharing ideas email firstname.lastname@example.org
Contract testing is hard, but it doesn't need to be. Many large companies are starting to build their own live testing tools that contract test their APIs.
What's different this time around? Coverage! When you pair live-sampling with coverage you get the confidence that your API does what you say it does. Live testing is effective, simple and demands much less work from engineers. Optic's team is really excited to be leading this charge and plan to deliver live-testing publicly in early 2020.
What are your thoughts? Is live-testing a suitable alternative to traditional contract testing? What questions or concerns do you have?