Skip to content

fix: Add timeouts to outgoing requests - #118

Open
billyvg wants to merge 6 commits into
mainfrom
fix/request-timeouts
Open

billyvg wants to merge 6 commits into
mainfrom
fix/request-timeouts

Conversation

@billyvg

@billyvg billyvg commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

No outgoing request had a timeout. gunicorn runs with 1 worker, 4 threads and --timeout 0 (see the Dockerfile), so a single hung connection to GitHub or Sentry could tie up a quarter of the app's capacity indefinitely.

Request latency over the last 7 days, from this app's spans:

Call Count p99 Max
GET config (.sentry/sentry_config.ini) 2.85M 165 ms 235 s
GET workflow run 1.08M 390 ms 127 s
POST installation access token 2.85M 229 ms 198 s
DELETE installation token 2.85M 308 ms 205 s
POST Sentry envelope 1.06M 70 ms 5.4 s

Changes

  • REQUEST_TIMEOUT = (5, 10), i.e. 5 s to connect and 10 s between bytes read, in src/__init__.py. It's passed to all five requests calls in src/.
    • That's over 25× the slowest p99, so normal traffic isn't affected.
    • The multi-minute stalls now fail fast with requests.exceptions.Timeout. That's handled like any other request error: logged, and the webhook gets a 500.
    • GitHub stops waiting for a webhook response after 10 s anyway.
  • cli.py is unchanged. It's a local development tool and isn't in the Docker image.
  • Only send the installation token to GitHub's API. _fetch_github rejects any URL that isn't under https://api.github.com/, and builds the request on that base.
  • Check the org login against GitHub's rules (alphanumerics and hyphens, up to 39 characters) before building the config URL.

#117 has since merged, and main is merged into this branch. The timeout and the login check are in _fetch_dsn, behind #117's DSN cache.

Testing

  • New test test_every_request_has_a_timeout drives a full webhook through WebAppHandler in GitHub App mode: mint token, fetch config, fetch run, post envelope, revoke token. It asserts that each request was sent with REQUEST_TIMEOUT.
  • _fetch_github refuses another host, http, a lookalike host, a URL with userinfo, and github.com, without making a request.
  • A 39-character login is accepted. A 40-character one, a leading hyphen, a path, an empty login, or one ending in a newline is rejected without a request.
  • Mutation check: removing the timeout from any one of the five call sites makes the test fail and name that URL. Putting back the 38-character limit, checking the login with match instead of fullmatch, or dropping the api.github.com check makes its test fail too.
  • pytest on the merged tree: 35 passed, 1 skipped (already skipped before this change).
  • pre-commit run passes on all changed files.

CodeQL

The two py/full-ssrf alerts are fixed by this PR. They're on the GETs whose URLs come from the webhook payload: the config fetch, and _fetch_github.

  • The config fetch only runs for a valid GitHub login, which can't change the URL's host or path.
  • _fetch_github builds the request on https://api.github.com, so the token can't be sent to another host.

Running CodeQL 2.27.2's default Python suite locally, it reports py/full-ssrf at src/github_sdk.py:60 before this change and nothing after it. Both alerts already exist on main, so merging this should also close #1 there.

🤖 Generated with Claude Code

No request had a timeout, and gunicorn runs with one worker, 4 threads
and `--timeout 0`, so a hung connection to GitHub or Sentry tied up a
thread indefinitely. Over the last 7 days the GitHub calls had a p99 of
165-390 ms but a max of 127-235 s, and Sentry ingest a p99 of 70 ms
with a max of 5.4 s.

Every request now uses a (5 s connect, 10 s read) timeout, which is
well above normal latency. GitHub also stops waiting for a webhook
response after 10 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/github_sdk.py Fixed
Comment thread src/sentry_config.py Fixed
billyvg and others added 2 commits October 8, 2026 17:15
…uest forgery'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…uest forgery'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@billyvg
billyvg marked this pull request as ready for review October 8, 2026 21:16
@billyvg
billyvg requested a review from a team as a code owner October 8, 2026 21:16
@billyvg
billyvg requested a review from armenzg October 8, 2026 21:17
Comment thread src/sentry_config.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 446b3a2. Configure here.

Comment thread src/sentry_config.py
billyvg and others added 2 commits October 9, 2026 14:09
Addresses review findings and the failing CodeQL check:

- GITHUB_OWNER_PATTERN allowed 38 characters, but GitHub logins can be
  39, so webhooks from such an owner failed with ValueError.
- CodeQL still reported py/full-ssrf on _fetch_github, because the
  host checks in a separate function don't sanitize the URL that's
  requested. Only accept URLs under https://api.github.com/, and build
  the request URL on that constant base, which CodeQL recognizes. The
  token was also allowed to go to github.com, which _fetch_github never
  needs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves the conflict with #117 in src/sentry_config.py: the request
timeout and owner check now live in _fetch_dsn, behind #117's DSN cache.

- #117's tests used `other_org`, which isn't a valid GitHub login, so
  the owner check rejected it. They now use `other-org`.
- test_every_request_has_a_timeout no longer registers the workflow
  fetch, which #117 removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/sentry_config.py

SENTRY_CONFIG_API_URL = (
"https://api.github.com/repos/{owner}/.sentry/contents/sentry_config.ini"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The regex validation for the org parameter uses $ which incorrectly matches strings with a trailing newline, potentially causing downstream API requests to fail.
Severity: LOW

Suggested Fix

Replace the $ anchor in the regex pattern with \Z. The \Z anchor ensures that the match must occur at the absolute end of the string, correctly rejecting any input with trailing newlines.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry_config.py#L22

Potential issue: The regex validation for the GitHub organization name uses a `$`
anchor, which allows `re.fullmatch()` to successfully match strings that have a trailing
newline, such as `"myorg\n"`. If the `org` parameter from a webhook payload contained
such a newline, it would pass validation. The subsequent call to `quote(org, safe="")`
would URL-encode the newline to `%0A`, creating an invalid GitHub API URL. This would
cause the API request to fail, likely with a 404 or 422 error, leading to an unhandled
`HTTPError` and a 500 response from the service. While it is unlikely that a GitHub
webhook would contain a trailing newline, the validation logic is incorrect.

A review suggested `$` lets `"myorg\n"` through. It doesn't, because the
check uses fullmatch, but only fullmatch prevents it: with match, the
login would pass. Cover it so the check can't regress to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants