Every test is green. The deploy goes out. Within an hour something downstream is throwing errors on data it has consumed happily for months.
This is one of the most common failure modes in API work, and it is rarely a bug in the tests. It is a gap in what they were written to check.
The usual shape of the problem
Most API test suites assert two things: the status code, and a few fields the test author cared about.
expect(res.status).toBe(200)
expect(res.body.id).toBe(42)
expect(res.body.name).toBe("Widget")
That test passes when the response is exactly what was expected. It also passes when the response is this:
{
"id": 42,
"name": "Widget",
"price": "19.99",
"tags": null,
"supplier": { "id": 7 }
}
…even though price just changed from a number to a string, tags became null where it
used to be an array, and supplier lost the name field that a different service depends
on. Nothing asserted on those, so nothing failed. The API is, from your test suite's point
of view, working perfectly.
That is contract drift: the API's actual shape moving away from what its consumers assume, without any error being raised on the way.
Why a 200 tells you almost nothing
We covered this in why 200 OK is not enough, but it is worth restating in the CI context: a status code describes what happened to the request, not whether the body is what you agreed on. An API that returns a correctly-formed 200 containing the wrong shape is, mechanically, a success.
The failure modes that slip through green suites are consistently these:
| Change | Why tests miss it |
|---|---|
Number becomes a string (19.99 → "19.99") | Loose equality and JSON parsing hide it |
| Field becomes nullable | The test data happened to be non-null |
| Field removed | Nothing asserted on it |
| Enum gains a value | The test only ever saw the old values |
| Array becomes a single object | Only checked when the list had one item |
| Timestamp format changes | String comparison still "works" |
Each is invisible to a status-code check and to any assertion that does not mention the affected field.
Check the whole shape, not the fields you remembered
The fix is to stop asserting on fields and start asserting on the contract. Instead of listing what you expect to be true, describe what a valid response is, and check the whole body against it.
That means a schema — JSON Schema, an OpenAPI response definition, or an equivalent — that
says: price is a number, tags is an array of strings, supplier has id and name, and
no unexpected extra fields are silently tolerated.
The difference in practice:
- A field-level assertion fails when the value you checked changes.
- A schema check fails when anything about the shape changes.
The second is what you want, because the field that breaks production is reliably the one nobody wrote an assertion for.
Getting a schema when there is no spec
The obvious objection: "we do not have an OpenAPI spec, and the third-party API we depend on certainly does not."
Two routes out of that.
If a spec exists, import it. An OpenAPI 3.x or Swagger document already contains response schemas. You can validate live responses against an OpenAPI spec directly, and importing the spec gets you the schemas without writing them.
If no spec exists, generate one from reality. Capture real responses, derive a schema from what the API actually returns, then tighten it by hand — mark the fields that must never be null, narrow the enums, fix the types the generator guessed loosely. This gives you a contract for an API that never published one, which is the common case with third-party services. SchemaClient can build a schema from captured traffic in the monitor for exactly this reason.
Generated schemas start permissive. That is fine — a permissive schema still catches a type change, and you tighten it as you learn the API.
Run the check in CI, against something real
A contract check that only runs on a developer's machine catches drift the week someone happens to look. To catch it the day it happens, it has to run on a schedule, in CI, like any other test.
Two things make this materially more useful:
Run against a live or staging environment, not a mock. A mock returns what you told it to return, which encodes the very assumption you are testing. Contract drift is a disagreement between your belief and reality, and a mock only ever reflects your belief.
Fail the build, do not just log. A warning in a CI log is a warning nobody reads. If the contract broke, the pipeline should go red.
SchemaClient's CLI runs collections and checks responses against schemas from a pipeline —
@schemaclient/cli is on npm and the CI runner guide covers wiring it into a
workflow. Whatever tool you use, the shape of the job is the same: send the requests, check
the responses against the contract, exit non-zero on a mismatch.
What good looks like
A team that has closed this gap can answer three questions immediately:
- What shape does each endpoint actually return today? There is a schema, and it is checked rather than aspirational.
- When did it last change? The build that went red tells you, with the failing field named.
- Who depends on the part that changed? Because the contract is written down, this is a search rather than an archaeology project.
None of that requires rewriting your test suite. It requires adding one check that looks at the whole response instead of the three fields somebody remembered to assert on — and running it where a failure is impossible to ignore.