Verify OpenAPI Accuracy

One of the most frustrating things for consumers is when an API is not behaving as documented. No team sets out to ship versions of their API that are out of sync with the documentation -- it happens because verifying API behavior is hard. With Optic you can verify the API behavior in CI and understand your API Test Coverage (the % of your API functionality your testing covered). If optic verify detects no diffs, and you have high coverage, you can be very confident your API is working as-designed.

1. Capturing traffic from the correct environment

When verifying API behavior, it is important to verify collected traffic against the correct version of your OpenAPI file. We suggest running verify as part of your CI process, so it is difficult for your API to drift. Running verify against deployed environments is still useful and will catch API drift, but since those issues have already been released it is less valuable.

Optic can collect traffic from subprocesses to help you capture a lot of traffic quickly:

Running with your tests. Note: If your tests do not actually hit the network, Optic will not be able to see them. You may need to change how they run, or export a HAR from your test suite instead. We can help with this (set up office hours) (opens in a new tab).

optic capture openapi.yaml http://api.local.dev --command 'go test'

Many teams use a combination of API testing tools like Postman, their code tests, a fuzzer, etc. Others have a manual QA phase that will result in the usage of a lot of the API surface area. Optic lets you join these captures into one big capture by running them sequentially:

optic capture openapi.yaml http://api.local.dev --command 'go test'    # Go Tests
optic capture openapi.yaml http://api.local.dev --command 'newman run' # Postman's Newman runner
optic capture openapi.yaml http://api.local.dev                        # A manual capture

2. Run verify

This command compares the traffic you collected to your API's actual behavior.

optic verify openapi.yaml

In the example below Optic found one diff and provided a coverage report showing 72% of the API's functionality had been tested

» Verifying API behavior...
 
[ Diff ]  'site_admin' is not documented
operation: get /gists/{gistId}
154 | properties:
155 |   user:
156 |     type: object
157 |     properties:  Undocumented 'site_admin'
158 |       login:
159 |         type: string
160 |       id:
openapi.yaml
fix schema by running $ optic update
 
 
API Behavior Report
 
Total Requests          : 128
Diffs                   : 1
Undocumented operations : 0
Undocumented bodies     : 0
 
Coverage Report 72.7%
●●●●   get /users/{user}/followers
 60   200
●●◌◌   get /users/{user}/starred
 25
●●◌◌   get /gists/{gistId}
 17   200
●●◌◌   get /users/{user}/gists
 16   200
●◌◌◌   get /users
 10   200
◌◌◌◌   get /users/{user}/subscriptions
  0   200
 
OpenAPI and implementation are out of sync. Exiting 1

Since the process exits 1 when there are diffs, verify works great as a CI check that ensures your API and OpenAPI are in sync before every merge.

To resolve the issue above you can manually document site_admin or have Optic do it for you with optic update. After collecting traffic again and running verify there are no more diffs and the process exits 0.

 » Verifying API behavior...
 
 API Behavior Report
 
 Total Requests          : 128
 Diffs                   : 0
 Undocumented operations : 0
 Undocumented bodies     : 0
 
 Coverage Report 75.0%
 ●●●●   get /users/{user}/followers
   60   200
 ●●◌◌   get /users/{user}/starred
   25   200
 ●●◌◌   get /gists/{gistId}
   17   200
 ●●◌◌   get /users/{user}/gists
   16   200
 ●◌◌◌   get /users
   10   200
 ◌◌◌◌   get /users/{user}/subscriptions
    0   200
 
No diffs detected. OpenAPI and implementation appear to be in sync.

Share verified API documentation with Optic Cloud

Optic Cloud tracks accuracy of the API Documentation you put in the catalog to help teams trust each other's docs.

If you API is not being tracked in Optic Cloud, first add it:

optic login
optic api add openapi.yaml

When you run the verify command with the --upload flag, a badge will appear in your API docs

optic verify openapi.yaml --upload