Skip to content

test: check every response against the OpenAPI schema, and fix what it found - #253

Merged
aditeyabaral merged 14 commits into
pesu-dev:devfrom
aditeyabaral:test/openapi-contract-fuzz
Oct 17, 2026
Merged

aditeyabaral merged 14 commits into
pesu-dev:devfrom
aditeyabaral:test/openapi-contract-fuzz

Conversation

@aditeyabaral

@aditeyabaral aditeyabaral commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

📌 Description

Adds tests that check every route's real responses against the OpenAPI schema, for a wide variety of inputs, and fixes the six problems they turned up.

test_openapi_docs.py proves each route has docs and that their examples are valid. Until now nothing checked that what the routes actually answer matches those docs. tests/unit/test_openapi_contract.py does:

  • Schemathesis generates requests from the schema, valid and invalid.
  • Hypothesis generates the inputs the schema cannot describe.
  • Every request goes to the app in process, and the test fails on any response whose status, Content-Type or body the docs do not describe, or on any 500.

PESU Academy is faked at the HTTP layer (the existing wire fixture), so every login runs the real PESUAcademy code, offline. Every generated test is seeded or derandomized, so each run sends the same data. The file takes about 20 seconds.

What each route is sent

Route Requests per run What must hold
/authenticate: generated requests 1,084 (1,000 random, plus Schemathesis' examples and boundary cases) A documented response, never a 500. A 200 means PESU received exactly the username and password sent, and the username has only visible characters. A 400 or 405 means nothing was sent to PESU.
/authenticate: generated PESU profiles 1,000 Each field the profile reads holds its usual value, any other JSON value, or is left out, with a random choice of fields. The answer is a documented 200 with exactly the requested fields (null where unusable), or the documented 422.
/authenticate: explicit cases 11 Each way a login can fail, as PESU's real answer: 401, 502 (unavailable, unreachable or non-JSON login, or a failed profile call), 422 (no student) and 500. Also four bodies FastAPI cannot read, each a documented 400.
/metrics 250 (125 open, 125 behind METRICS_TOKEN), plus Schemathesis' examples Generated fmt values and Authorization headers get documented responses. When protected, no generated request gets metrics. Explicit cases cover rejected and accepted tokens.
/, /docs, /redoc, /openapi.json, /docs/oauth2-redirect, /health, /readme 200 each Any query string, under any method. GET gets the usual answer, HEAD the same without a body, and any other method a standard 405 whose Allow header lists exactly the path's methods.
Unknown paths 200 Any path, under any method, gets a standard 404.

A final test proves the checks really reject a broken response.

Fixes

  1. Log injection through the username (security). The username is logged as sent, by design. So {"username": "PES1UG20CS001\n2026-10-17 07:00:00,000 - ERROR - Login successful for user=admin", ...} wrote a forged ERROR line into the logs, and the username went on to PESU Academy unchanged.
    • No SRN, PRN, email or phone number contains a control or invisible character. A username with one is now a 400, before anything is logged. That covers anything str.isprintable() rejects: line breaks, escapes, NUL, U+2028, and zero-width and no-break spaces.
    • Spaces and other visible text are unchanged. Passwords are never logged and still accept anything.
    • The generated requests found this: without the fix, Schemathesis sends a username with a terminal escape that reaches PESU.
  2. Unknown campus names in the log. The warning for an unknown campus wrote PESU's name for it raw, so a name containing a line break would have forged log lines the same way. It is now written quoted, like the other values logged from PESU's responses.
  3. Unreadable request body. FastAPI raises a 400 HTTPException for JSON that is not valid UTF-8, rather than a RequestValidationError. The caller got {"detail": "There was an error parsing the body"} and the error was never counted. It is now a RequestBodyParseError: the standard 400 body with a fixed message, counted in errors_total. The unread bytes are never echoed or logged.
  4. 404 and 405 in the standard shape. An unknown path or an unsupported method got the framework's {"detail": ...}, the one exception to the documented {status, message, timestamp} body.
    • They are now RouteNotFoundError (404) and MethodNotAllowedError (405), rendered and counted like every other error, without echoing the path.
    • A 405's Allow header lists every method the path is served for. Starlette's would have left HEAD out.
    • Every failed response now goes through an exception handler, so sum(errorsByType) equals requests.failed.
  5. HEAD on the GET routes. HEAD /health, /readme and /metrics answered 405: FastAPI's GET routes answer GET only, unlike Starlette's, which serve the docs pages. Uptime monitors often check with HEAD, so one would have reported a healthy service as down.
    • Each route now has a HEAD route beside it, kept out of the schema, and the server sends the status and headers without the body. I checked this on a real uvicorn.
    • Two routes, rather than one with both methods, because one route would put duplicate operation IDs in the schema.
    • HEAD /metrics needs the token like GET.
  6. /metrics?fmt=json docs. The JSON response was documented with an empty schema. It now references MetricsModel, so the existing example check also validates the JSON example.

This replaces #228, starting again from current dev (the 5.x mobile API) and keeping only its in-process part. Left out:

  • the URL and CLI runs against a separate server on :8080;
  • the pre-commit hook and CI job for those runs;
  • TraceCov;
  • the Schemathesis-generated live logins. The fake at the HTTP layer now runs the same code offline.

Together they were most of #228's 1,600 lines, made six extra real logins to PESU Academy on every run, and needed an unsanitized-output config. The /readme 308 docs fix from #228 is already on dev.

ℹ️ Fixes / Related Issues
Fixes: #218
Related: #228

🧱 Type of Change

  • 🐛 Bug fix – Non-breaking fix for a functional/logic error
  • 🔒 Security fix – Addresses auth/session/data validation vulnerabilities
  • 📝 Documentation update – README, docstrings, OpenAPI tags, etc.
  • 🧪 Test suite change – Adds/updates unit, functional, or integration tests
  • 🧰 Dependency update – Updates libraries in pyproject.toml, uv.lock

🧪 How Has This Been Tested?

  • Unit Tests (tests/unit/)
  • Functional Tests (tests/functional/)
  • Integration Tests (tests/integration/)
  • Manual Testing

⚙️ Test Configuration:

  • OS: Linux
  • Python: 3.14 via uv
  • Docker build tested (runtime dependencies and Dockerfile unchanged)
  • uv run python scripts/run_tests.py with .env credentials: 628 passed, including all live tests, with 100% coverage.
  • uv run pre-commit run --all-files: all hooks pass.
  • The regression tests for each fix fail on dev and pass here.
  • Manual check: HEAD, 404 and 405 against a real uvicorn (below).
  • To confirm the generated tests catch real bugs, I planted five, one at a time. Each made a generated test fail:
    • the date-of-birth overflow guard removed;
    • the password stripped before it is sent;
    • /metrics accepting any bearer token;
    • an unknown campus raising instead of returning null;
    • the username check removed.
  • Larger, unseeded runs (5,000 /authenticate requests, 5,000 profiles, 3,000 /metrics requests) found no undocumented response or 500. They also found no password, access token or metrics token in any log line, and metric labels stayed bounded.

✅ Checklist

  • My code follows the CONTRIBUTING.md guidelines
  • I've performed a self-review of my changes
  • I've added/updated necessary comments and docstrings
  • I've updated relevant docs (README or endpoint docs)
  • No new warnings introduced
  • I've added tests to cover my changes
  • All tests pass locally (scripts/run_tests.py)
  • I've run linting and formatting (pre-commit run --all-files)
  • Changes are backwards compatible (if applicable): see the behaviour changes below

🛠️ Affected API Behaviour

  • app/app.py – Modified /authenticate route logic
  • app/pesu.py – Updated PESU Academy login or profile handling
  • app/exceptions/ – Error types, their status codes or messages
  • app/docs/ – OpenAPI (Swagger) examples and descriptions

🧩 Models

  • app/models/request.py – Input validation or request schema changes

🐳 DevOps & Config

  • pyproject.toml / uv.lock – Dependency version changes

📸 Screenshots / API Demos (if applicable)

$ curl -s -X POST localhost:5000/authenticate -H 'Content-Type: application/json' --data-binary $'\xff'
# before: {"detail":"There was an error parsing the body"}
{"status":false,"message":"Could not parse request body.","timestamp":"..."}

$ curl -s -I localhost:5000/health
# before: HTTP/1.1 405 Method Not Allowed
HTTP/1.1 200 OK
content-length: 77
content-type: application/json

$ curl -s -i -X POST localhost:5000/health
# before: allow: GET, and {"detail":"Method Not Allowed"}
HTTP/1.1 405 Method Not Allowed
allow: GET, HEAD
{"status":false,"message":"This method is not allowed on this path.","timestamp":"..."}

$ curl -s localhost:5000/nope
# before: {"detail":"Not Found"}
{"status":false,"message":"No route matches this path.","timestamp":"..."}

🧠 Additional Notes (if applicable)

  • Behaviour changes. I kept these to a minor bump (5.3.0), since neither breaks a request that could have worked:
    • A 404 or 405 body now has message where it had detail.
    • A username with an invisible character is now a 400. PESU Academy could never have matched it to a real account.
    • Every 404 and 405 is also counted in errors_total and logged at WARNING with a fixed message, like other client errors. Scanners probing for other software show up there.
  • Request counts. Each test sets its own count. The fake PESU Academy accepts every username (SRN, PRN, email or phone), so about a third of the generated /authenticate requests reach the 200 path. Every failure has its own explicit case.
  • Failing runs. A test that fails can take a few minutes to finish, because Hypothesis shrinks the failing input to the smallest one. A passing run is unaffected.
  • Dependencies. schemathesis (MIT) and hypothesis are dev-only. schemathesis is capped below 5, because its Python API changes between majors.
  • New responses. A new status, Content-Type or response shape now needs its app/docs/ entry first, or the contract test fails.

🤖 Generated with Claude Code

FastAPI raises a 400 HTTPException, not a RequestValidationError, when it
cannot read a request body at all, such as JSON that is not valid UTF-8.
That bypassed this API's handlers: the caller got FastAPI's
{"detail": ...} body instead of the documented {status, message,
timestamp}, and the error was never counted in errors_total. Render it as
a RequestBodyParseError, without the unread bytes, and leave the
framework's other errors (404, 405) as they were.

/metrics?fmt=json was also documented with an empty schema; it now
references MetricsModel, so its example is checked against the model too.

Both were found by fuzzing the OpenAPI contract in pesu-dev#228.
test_openapi_docs.py proves each route has docs and that their examples
are valid, but nothing checked that the routes actually answer what the
docs say. Schemathesis now generates requests from the schema, valid and
invalid, for every operation, sends them to the app in process with PESU
Academy faked, and fails on any response whose status, Content-Type or
body the schema does not document. Seeded, with no example database, so
every run sends the same requests.

Explicit cases cover what generation cannot reach: each /authenticate
failure (401, 422, 500, 502), bodies FastAPI cannot read, and /metrics
behind a token. A last test proves the checks reject a broken response,
so they cannot silently pass everything.

This takes the in-process part of pesu-dev#228. The URL and CLI runs against a
separate server, the pre-commit hook and CI job for them, TraceCov, and
the Schemathesis-generated live logins are left out: they need a lot of
machinery and extra logins on the single-session test account, and add
little over this test for four routes with strict models.
Minor: an unreadable request body now gets the documented 400 body instead of FastAPI's, and the OpenAPI contract is tested.
@aditeyabaral
aditeyabaral requested a review from a team as a code owner October 16, 2026 16:20
Copilot AI balanced review requested due to automatic review settings October 16, 2026 16:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@jois-code
jois-code requested a lite review from Copilot October 16, 2026 16:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

The 500 case raised a PESUAcademyError from authenticate, which it never
raises, because an unhandled exception is re-raised by the server after
the 500 is rendered and failed the test before the response was checked.
Send the failures through a TestClient that returns the response instead,
so the case is a real unhandled exception reaching the catch-all.

pesu-dev#218 asks for a documented way to run a deeper fuzz locally. Register a
Hypothesis profile, so --hypothesis-profile=deep sends 1,000 requests per
route instead of the default 100, and say so in CONTRIBUTING.
/authenticate is what callers use this service for, so its generated test
gets five times the budget of the other routes: 500 requests by default,
5,000 with --hypothesis-profile=deep. It adds about 2 seconds to the unit
suite. The other routes keep 100 each.
The contract test replaced PESUAcademy.authenticate, so generated requests
never ran the code that sends credentials to PESU Academy or builds the
profile. Since 5.0 a login is two plain POSTs with no CSRF state, so the
wire fixture can stand in for PESU Academy instead, and every generated
login now runs the real PESUAcademy.

/authenticate: generated requests also check that the credentials reach
PESU Academy exactly as sent, and that a request turned away never
reaches it. A new test generates PESU's profile response, with any JSON
value or none in each field the profile reads, and any choice of fields.
The explicit failure cases are now PESU's own answers on the wire.

/metrics: generated requests run both open and behind METRICS_TOKEN,
where none of them may get metrics without the token.

Pages without input (the docs pages, /health, /readme): any query string,
under any method, gets the usual answer or a 405 with the right Allow.

The multipart decoder moves to conftest and keeps values exactly, so a
password ending in a line break round-trips.
Every route without input now gets 200 generated requests of its own:
the page test runs once per page, rather than sharing 100 requests
across all seven. /metrics gets 250 in each of its open and protected
runs, and /authenticate keeps 500.
…ected runs

The user asked for 250 requests to /metrics overall, not 250 per run, so each of the two runs now sends 125.
Each budget was a multiple of Hypothesis' default, so that a "deep"
profile could scale them all. That made the numbers hard to read and the
deep run is not needed, so each test now states its own count (500 for
/authenticate, 125 for each /metrics run, 200 for each other route) and
the profile is gone.
/authenticate is the endpoint callers depend on, so both its generated tests -- the requests callers send and the profiles PESU Academy answers with -- now get 1000 requests each.
The username is logged by design, as it was sent. A username holding a
line break therefore wrote lines of its own into the logs: one request
could add a forged "ERROR ... Login successful for user=admin" line. It
was also passed on to PESU Academy as it was.

No SRN, PRN, email or phone number has a control or invisible character,
so a username with one (str.isprintable() false: line breaks, escapes,
NUL, U+2028, zero-width and no-break spaces) is now a 400 before
anything logs it. Spaces and other visible text are unchanged. Passwords
are not logged and keep accepting anything.

Found by the generated /authenticate requests: the contract test now
also checks that only a username of visible characters reaches PESU
Academy, and fails without this fix.
/health, /readme and /metrics answered HEAD with a 405: FastAPI's GET
routes answer GET only, unlike Starlette's, which the docs pages use.
Uptime monitors often check with HEAD, so one would have reported a
healthy service as down. Each now has a HEAD route beside its GET route,
kept out of the schema; the server sends the status and headers without
the body. Two routes rather than one with both methods, which would put
two operations with the same ID in the schema. HEAD /metrics needs the
token like GET.

An unknown path or an unsupported method got the framework's
{"detail": ...} body, the one exception to the documented
{status, message, timestamp} shape. They are now RouteNotFoundError (404)
and MethodNotAllowedError (405), rendered and counted like every other
error, without echoing the path. A 405's Allow header lists every method
the path is served for: Starlette's names only the first route's, which
would have left HEAD out. Any other HTTPException the framework raises
also gets the standard shape.

Every failed response now goes through an exception handler, so
sum(errorsByType) equals requests.failed; the README said 404s and 405s
were missing from it.
The warning for an unknown campus wrote PESU Academy's name for it as
is, so a name holding a line break would have added lines of its own
to the log, the same as a username could. It is now written quoted, as
the other values logged from PESU's responses already are.

Also gives the direct HTTPException-handler test a collector of its own,
since the handler counts the error, and corrects the comment on the
authentication failure cases, one of which is our own code failing.
@aditeyabaral aditeyabaral changed the title test: check every response against the OpenAPI schema test: check every response against the OpenAPI schema, and fix what it found Oct 17, 2026
Comment thread .github/CONTRIBUTING.md Outdated
Comment thread app/models/request.py Outdated
Comment thread app/app.py Outdated
Comment thread app/app.py Outdated
Comment thread README.md Outdated
Drops the contract test's bullet from CONTRIBUTING, trims the HEAD-route and metrics-token comments, and takes "visible" back out of the username's description in the schema and the README.
@aditeyabaral
aditeyabaral added this pull request to the merge queue Oct 17, 2026
Merged via the queue into pesu-dev:dev with commit be6a0a4 Oct 17, 2026
7 checks passed
@aditeyabaral
aditeyabaral deleted the test/openapi-contract-fuzz branch October 17, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fuzz the API against its OpenAPI schema with Schemathesis

3 participants