Skip to content

feat: x402 crawl gateway for training crawlers - #318

Merged
ralyodio merged 1 commit into
masterfrom
feat/crawl-gateway
Oct 21, 2026
Merged

ralyodio merged 1 commit into
masterfrom
feat/crawl-gateway

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

Wires @profullstack/x402-gateway 0.11 into src/middleware.js (now async), after the www redirect and before rate limiting.

  • GPTBot and other training crawlers: 402 with PAYMENT-REQUIRED
  • /crawl: $1/day pass offer (USDC on Base/Polygon/Ethereum) plus the CoinPay "Buy a day pass" checkout
  • Browsers, verified Googlebot, AI chat fetchers: unchanged
  • Open: /api/ (health, auth hooks, webhooks), /ws, /_next/, manifest, sw.js, robots, sitemap
  • Env read at runtime (COINPAY_X402_KEY, CRAWL_PAY_TO), in dev2 app.env and the qryptchat-web prod vault

Test: tests/crawl-gateway.test.js; full vitest run 118 files / 764 tests pass.

🤖 Generated with Claude Code

Training crawlers get a 402 with PAYMENT-REQUIRED and /crawl sells a $1 day
pass (x402 USDC or CoinPay checkout). Browsers, verified search crawlers and
AI chat fetchers pass through; /api/, /ws and static assets stay open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

14 finding(s)

MEDIUM: 10 | LOW: 4

Severity Rule Location
MEDIUM js-unescaped-html-sink src/app/blog/[slug]/page.jsx:38
MEDIUM js-unescaped-html-sink src/app/blog/[slug]/page.jsx:66
MEDIUM js-unescaped-html-sink src/app/faq/page.jsx:57
MEDIUM js-unescaped-html-sink src/app/layout.jsx:142
MEDIUM js-unescaped-html-sink src/app/layout.jsx:146
MEDIUM js-unescaped-html-sink src/app/page.jsx:49
MEDIUM redos-nested-quantifier src/lib/auth/dns-name.js:88
MEDIUM js-unescaped-html-sink src/lib/components/chat/MessageItem.jsx:85
MEDIUM js-unescaped-html-sink src/lib/components/chat/MessageItem.jsx:130
MEDIUM js-unescaped-html-sink src/lib/components/PitFooter.jsx:15
LOW secret-generic-credential src/app/api/auth/register-anon/route.test.js:32
LOW secret-jwt tests/debug-sms.js:10
LOW secret-generic-credential tests/private-key-import-export.test.js:252
LOW secret-generic-credential tests/private-key-import-export.test.js:264

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio
ralyodio merged commit b16e4b0 into master Oct 21, 2026
8 checks passed
@ralyodio
ralyodio deleted the feat/crawl-gateway branch October 21, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant