OpenAPI-Pact-Comparator

What is this?

This compares OpenAPI schemas and Pact contracts to determine if they are compatible. It is inspired by swagger-mock-validator and aims to retain a mostly compatible interface.

Why rewrite it?

swagger-mock-validator has aged, and is very difficult to extend and improve on.

It is also very slow primarily due to inefficient use of ajv; schemas are unnecessarily recompiled everytime instead of being cached.

How is this better?

Schemas are compiled once, and reused where possible. In practical terms, this gives us an improvement exceeding 20x in real-world comparisons. In some cases, this was 50x faster!

Referenced schemas are NOT inlined. They are kept as references to keep the size of complex schemas down. We need to do this anyway to support circular references. Further, unused references are skipped to speed up schema compilation in AJV.

Multiple pacts can be compared in one invocation to maximise schema reuse. Instead of performing the comparison per pair of OAS + Pact, we can reuse the compiled OAS schemas across multiple Pacts.

A fast HTTP router is used to match provider routes. Instead of iterating through an array of routes, we use Radix Tree search using find-my-way This allows large providers to be traversed quickly.

Computation is broken up to small units using async generators. This leads to low event loop delays, suitable for high concurrency servers.

Usage

With error handling omitted for brevity:

import { Comparator } from "openapi-pact-comparator";

// oas/asyncapi is object from JSON.parse() or yaml.load()
const comparator = new Comparator({ oas, asyncapi });

// pacts is array of objects the same way
for (const pact of pacts) {
  for await (const result of comparator.compare(pact)) {
    console.log(result);
  }
}

Avro and Protobuf

Avro and Protobuf message payloads are supported for AsyncAPI, using Pact v4 files from pact-avro-plugin and pact-protobuf-plugin. Protobuf well-known types such as google/protobuf/timestamp.proto can be imported. Other imports cannot be resolved from a single .proto source, so an AsyncAPI schema that uses them is reported as unsupported rather than compared. A pact's own schema is read from the descriptors the plugin embeds, which already include its imports.

The consumer's payload and the provider's message must be the same format. An Avro or Protobuf payload against a message declared as anything else, or a plain (for example JSON) payload against an Avro or Protobuf message, fails with message.payload.format.incompatible.

Each interaction is compared in two independent passes, and each reports what it finds:

• Values. The consumer's example is decoded and validated against the provider's schema converted to JSON Schema, as for JSON payloads.
• Structure. The consumer's own schema, which the plugin embeds in the pact, is compared with the provider's, the way the consumer's decoder will read the provider's messages: Protobuf by field and enum number (only the fields and enum values the consumer uses), Avro by writer/reader schema resolution (assuming consumers decode with the writer's schema, e.g. via a schema registry). This catches changes that keep the JSON shape but break the consumer, such as a renumbered field, int32 to sint32, a renumbered enum, int to long, a new enum symbol, a field made nullable, or a renamed record.

The same problem can therefore be reported by both passes. The structural pass also reports message.payload.type.incompatible when the consumer's and provider's messages are different types (the Protobuf message, or Avro record, the pact declares is not the one the provider's message carries). It stays silent when it cannot tell, for example when a schema cannot be parsed, and the value pass explains why the payload was not validated.

The provider's Protobuf message is chosen by a proto content type parameter, the payload's x-drift-proto-message extension, the message's key or name, or the only message in the schema, in that order.

Payloads are decoded to JavaScript numbers, which can only represent integers exactly up to 2^53 (9,007,199,254,740,992). Beyond that:

• Protobuf int64, uint64, sint64, fixed64 and sfixed64 values, and Avro long values, are silently rounded to the nearest representable number. The comparison still runs, so a rounded value could pass or fail a check it would not have with the exact value.

Quirks mode

To retain compatibility with swagger-mock-validator, an environment variable QUIRKS can be set [to any value]. When this is true, all of SMV quirks are reproduced in case you are unable to migrate immediately.

The quirks can also be enabled/disabled by adding to the info section of the OAS some extensions in the form of x-opc-config-${quirk-name}, where quirks are listed here.

This mode may be removed eventually, so you should always endeavour to try to update your comparisons to use the latest functionality.