Repository navigation
test: check every response against the OpenAPI schema, and fix what it found - #253
Merged
aditeyabaral merged 14 commits intoOct 17, 2026
Merged
Conversation
FastAPI raises a 400 HTTPException, not a RequestValidationError, when it
cannot read a request body at all, such as JSON that is not valid UTF-8.
That bypassed this API's handlers: the caller got FastAPI's
{"detail": ...} body instead of the documented {status, message,
timestamp}, and the error was never counted in errors_total. Render it as
a RequestBodyParseError, without the unread bytes, and leave the
framework's other errors (404, 405) as they were.
/metrics?fmt=json was also documented with an empty schema; it now
references MetricsModel, so its example is checked against the model too.
Both were found by fuzzing the OpenAPI contract in pesu-dev#228.
test_openapi_docs.py proves each route has docs and that their examples are valid, but nothing checked that the routes actually answer what the docs say. Schemathesis now generates requests from the schema, valid and invalid, for every operation, sends them to the app in process with PESU Academy faked, and fails on any response whose status, Content-Type or body the schema does not document. Seeded, with no example database, so every run sends the same requests. Explicit cases cover what generation cannot reach: each /authenticate failure (401, 422, 500, 502), bodies FastAPI cannot read, and /metrics behind a token. A last test proves the checks reject a broken response, so they cannot silently pass everything. This takes the in-process part of pesu-dev#228. The URL and CLI runs against a separate server, the pre-commit hook and CI job for them, TraceCov, and the Schemathesis-generated live logins are left out: they need a lot of machinery and extra logins on the single-session test account, and add little over this test for four routes with strict models.
Minor: an unreadable request body now gets the documented 400 body instead of FastAPI's, and the OpenAPI contract is tested.
21 of 24 tasks
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
The 500 case raised a PESUAcademyError from authenticate, which it never raises, because an unhandled exception is re-raised by the server after the 500 is rendered and failed the test before the response was checked. Send the failures through a TestClient that returns the response instead, so the case is a real unhandled exception reaching the catch-all. pesu-dev#218 asks for a documented way to run a deeper fuzz locally. Register a Hypothesis profile, so --hypothesis-profile=deep sends 1,000 requests per route instead of the default 100, and say so in CONTRIBUTING.
/authenticate is what callers use this service for, so its generated test gets five times the budget of the other routes: 500 requests by default, 5,000 with --hypothesis-profile=deep. It adds about 2 seconds to the unit suite. The other routes keep 100 each.
The contract test replaced PESUAcademy.authenticate, so generated requests never ran the code that sends credentials to PESU Academy or builds the profile. Since 5.0 a login is two plain POSTs with no CSRF state, so the wire fixture can stand in for PESU Academy instead, and every generated login now runs the real PESUAcademy. /authenticate: generated requests also check that the credentials reach PESU Academy exactly as sent, and that a request turned away never reaches it. A new test generates PESU's profile response, with any JSON value or none in each field the profile reads, and any choice of fields. The explicit failure cases are now PESU's own answers on the wire. /metrics: generated requests run both open and behind METRICS_TOKEN, where none of them may get metrics without the token. Pages without input (the docs pages, /health, /readme): any query string, under any method, gets the usual answer or a 405 with the right Allow. The multipart decoder moves to conftest and keeps values exactly, so a password ending in a line break round-trips.
Every route without input now gets 200 generated requests of its own: the page test runs once per page, rather than sharing 100 requests across all seven. /metrics gets 250 in each of its open and protected runs, and /authenticate keeps 500.
…ected runs The user asked for 250 requests to /metrics overall, not 250 per run, so each of the two runs now sends 125.
Each budget was a multiple of Hypothesis' default, so that a "deep" profile could scale them all. That made the numbers hard to read and the deep run is not needed, so each test now states its own count (500 for /authenticate, 125 for each /metrics run, 200 for each other route) and the profile is gone.
/authenticate is the endpoint callers depend on, so both its generated tests -- the requests callers send and the profiles PESU Academy answers with -- now get 1000 requests each.
The username is logged by design, as it was sent. A username holding a line break therefore wrote lines of its own into the logs: one request could add a forged "ERROR ... Login successful for user=admin" line. It was also passed on to PESU Academy as it was. No SRN, PRN, email or phone number has a control or invisible character, so a username with one (str.isprintable() false: line breaks, escapes, NUL, U+2028, zero-width and no-break spaces) is now a 400 before anything logs it. Spaces and other visible text are unchanged. Passwords are not logged and keep accepting anything. Found by the generated /authenticate requests: the contract test now also checks that only a username of visible characters reaches PESU Academy, and fails without this fix.
/health, /readme and /metrics answered HEAD with a 405: FastAPI's GET
routes answer GET only, unlike Starlette's, which the docs pages use.
Uptime monitors often check with HEAD, so one would have reported a
healthy service as down. Each now has a HEAD route beside its GET route,
kept out of the schema; the server sends the status and headers without
the body. Two routes rather than one with both methods, which would put
two operations with the same ID in the schema. HEAD /metrics needs the
token like GET.
An unknown path or an unsupported method got the framework's
{"detail": ...} body, the one exception to the documented
{status, message, timestamp} shape. They are now RouteNotFoundError (404)
and MethodNotAllowedError (405), rendered and counted like every other
error, without echoing the path. A 405's Allow header lists every method
the path is served for: Starlette's names only the first route's, which
would have left HEAD out. Any other HTTPException the framework raises
also gets the standard shape.
Every failed response now goes through an exception handler, so
sum(errorsByType) equals requests.failed; the README said 404s and 405s
were missing from it.
The warning for an unknown campus wrote PESU Academy's name for it as is, so a name holding a line break would have added lines of its own to the log, the same as a username could. It is now written quoted, as the other values logged from PESU's responses already are. Also gives the direct HTTPException-handler test a collector of its own, since the handler counts the error, and corrects the comment on the authentication failure cases, one of which is our own code failing.
aditeyabaral
commented
Oct 17, 2026
Drops the contract test's bullet from CONTRIBUTING, trims the HEAD-route and metrics-token comments, and takes "visible" back out of the username's description in the schema and the README.
aditeyabaral
enabled auto-merge
October 17, 2026 02:37
aditeyabaral
requested review from
abaksy,
achyu-dev,
jois-code and
ndigvijay
October 17, 2026 02:41
abaksy
approved these changes
Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 Description
Adds tests that check every route's real responses against the OpenAPI schema, for a wide variety of inputs, and fixes the six problems they turned up.
test_openapi_docs.pyproves each route has docs and that their examples are valid. Until now nothing checked that what the routes actually answer matches those docs.tests/unit/test_openapi_contract.pydoes:PESU Academy is faked at the HTTP layer (the existing
wirefixture), so every login runs the realPESUAcademycode, offline. Every generated test is seeded or derandomized, so each run sends the same data. The file takes about 20 seconds.What each route is sent
/authenticate: generated requests/authenticate: generated PESU profilesfields. The answer is a documented 200 with exactly the requested fields (null where unusable), or the documented 422./authenticate: explicit cases/metricsMETRICS_TOKEN), plus Schemathesis' examplesfmtvalues andAuthorizationheaders get documented responses. When protected, no generated request gets metrics. Explicit cases cover rejected and accepted tokens./,/docs,/redoc,/openapi.json,/docs/oauth2-redirect,/health,/readmeAllowheader lists exactly the path's methods.A final test proves the checks really reject a broken response.
Fixes
{"username": "PES1UG20CS001\n2026-10-17 07:00:00,000 - ERROR - Login successful for user=admin", ...}wrote a forged ERROR line into the logs, and the username went on to PESU Academy unchanged.str.isprintable()rejects: line breaks, escapes, NUL, U+2028, and zero-width and no-break spaces.HTTPExceptionfor JSON that is not valid UTF-8, rather than aRequestValidationError. The caller got{"detail": "There was an error parsing the body"}and the error was never counted. It is now aRequestBodyParseError: the standard 400 body with a fixed message, counted inerrors_total. The unread bytes are never echoed or logged.{"detail": ...}, the one exception to the documented{status, message, timestamp}body.RouteNotFoundError(404) andMethodNotAllowedError(405), rendered and counted like every other error, without echoing the path.Allowheader lists every method the path is served for. Starlette's would have left HEAD out.sum(errorsByType)equalsrequests.failed.HEAD /health,/readmeand/metricsanswered 405: FastAPI's GET routes answer GET only, unlike Starlette's, which serve the docs pages. Uptime monitors often check with HEAD, so one would have reported a healthy service as down.HEAD /metricsneeds the token like GET./metrics?fmt=jsondocs. The JSON response was documented with an empty schema. It now referencesMetricsModel, so the existing example check also validates the JSON example.This replaces #228, starting again from current
dev(the 5.x mobile API) and keeping only its in-process part. Left out::8080;Together they were most of #228's 1,600 lines, made six extra real logins to PESU Academy on every run, and needed an unsanitized-output config. The
/readme308 docs fix from #228 is already ondev.🧱 Type of Change
pyproject.toml,uv.lock🧪 How Has This Been Tested?
tests/unit/)tests/functional/)tests/integration/)uv run python scripts/run_tests.pywith.envcredentials: 628 passed, including all live tests, with 100% coverage.uv run pre-commit run --all-files: all hooks pass.devand pass here./metricsaccepting any bearer token;/authenticaterequests, 5,000 profiles, 3,000/metricsrequests) found no undocumented response or 500. They also found no password, access token or metrics token in any log line, and metric labels stayed bounded.✅ Checklist
scripts/run_tests.py)pre-commit run --all-files)🛠️ Affected API Behaviour
app/app.py– Modified/authenticateroute logicapp/pesu.py– Updated PESU Academy login or profile handlingapp/exceptions/– Error types, their status codes or messagesapp/docs/– OpenAPI (Swagger) examples and descriptions🧩 Models
app/models/request.py– Input validation or request schema changes🐳 DevOps & Config
pyproject.toml/uv.lock– Dependency version changes📸 Screenshots / API Demos (if applicable)
🧠 Additional Notes (if applicable)
messagewhere it haddetail.errors_totaland logged at WARNING with a fixed message, like other client errors. Scanners probing for other software show up there./authenticaterequests reach the 200 path. Every failure has its own explicit case.schemathesis(MIT) andhypothesisare dev-only.schemathesisis capped below 5, because its Python API changes between majors.app/docs/entry first, or the contract test fails.🤖 Generated with Claude Code