Notify IndexNow about deployed changes to public research pages

.github/workflows/indexnow.yml

@@ -0,0 +1,38 @@
+name: IndexNow changed-page notifications
+
+on:
+  # GitHub Pages managed deployments can emit a successful deployment_status
+  # after a site is live. This avoids notifying about a page before deployment.
+  deployment_status:
+  # Reliable fallback and explicit re-submission for a genuine page update.
+  workflow_dispatch:
+    inputs:
+      urls:
+        description: 'Recently changed site-relative or absolute URLs (comma-separated; empty = infer most recent deployment diff)'
+        required: false
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: indexnow-notifications
+  cancel-in-progress: false
+
+jobs:
+  notify:
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.deployment_status.state == 'success' &&
+       github.event.deployment.environment == 'github-pages')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Read deployed source
+        uses: actions/checkout@v5
+        with:
+          ref: ${{ github.event_name == 'deployment_status' && github.event.deployment.sha || github.sha }}
+          fetch-depth: 2
+      - name: Notify only changed public URLs
+        env:
+          INDEXNOW_URLS: ${{ github.event.inputs.urls }}
+        run: python3 scripts/indexnow.py --root . --urls "$INDEXNOW_URLS"

.github/workflows/site-audit.yml

@@ -10,6 +10,9 @@ on:
       - '_includes/**'
       - '_config.yml'
       - 'scripts/audit_site.py'
+      - 'scripts/indexnow.py'
+      - 'scripts/test_indexnow.py'
+      - '.github/workflows/indexnow.yml'
       - 'research.json'
       - 'publications.json'
       - 'robots.txt'
@@ -32,3 +35,5 @@ jobs:
           destination: ./_site
       - name: Check built pages, citations, and bilingual research data
         run: python3 scripts/audit_site.py _site
+      - name: Test IndexNow submission validation offline
+        run: python3 -m unittest discover -s scripts -p 'test_indexnow.py' -v

README.md

@@ -142,3 +142,47 @@ The internal source-audit file `docs/publication-source-audit.md` is kept in
 GitHub for the maintainers but explicitly excluded from the public Jekyll build.
 The Search Console verification HTML must stay at the repository root and keep
 its exact token, even though it is not a research page.
+
+## IndexNow URL notifications
+
+The site uses the existing `/sitemap.xml` for full URL discovery. **IndexNow
+is only a change notification** for participating search engines (Bing and
+partners); it is **not** a Google Search Console submission, ranking signal,
+or indexing guarantee.
+
+Configuration:
+- Ownership: a 32-hex `<key>.txt` file at the public site root. It contains
+  exactly the same public key. Do not delete it while IndexNow is configured.
+  The Jekyll audit checks this file and keeps it out of the sitemap.
+- `.github/workflows/indexnow.yml`: submits only on successful
+  `github-pages` deployment-status events, with an explicit manual
+  `workflow_dispatch` fallback. Neither a draft PR nor a failed deploy
+  should notify search engines.
+- `scripts/indexnow.py`: reads `git diff HEAD^ HEAD`, finds added/updated
+  and deleted Jekyll permalinks, and sends a JSON URL batch to
+  `https://api.indexnow.org/indexnow`. A shared layout or configuration
+  change legitimately affects many pages; in that case it reads the current
+  live sitemap. It validates that every URL belongs to this exact host.
+- `scripts/test_indexnow.py` and the main site audit validate the key and
+  URL logic offline, without sending HTTP requests from a PR build.
+
+**After merging**, first confirm the key file is served at the public
+`https://floatingcatty.github.io/<key>.txt` URL. Check **Actions → IndexNow
+changed-page notifications**. If your GitHub Pages deployment type does not
+emit the expected `deployment_status` event, click **Run workflow** and
+pass a comma-separated list of recently changed URLs such as
+`/cv/,/research/vbcsr/`. An empty manual list attempts to infer changed
+URLs from the latest commit. For dry-run locally:
+
+```bash
+python3 -m unittest discover -s scripts -p 'test_indexnow.py' -v
+python3 scripts/indexnow.py --dry-run --urls '/cv/,/zh/cv/'
+```
+
+A 200 means the notification was received; 202 means received but key
+verification is pending. Neither status proves that any URL is indexed.
+Do not repeatedly resubmit the entire sitemap; notify meaningful changes
+only. Validate crawl/citation performance separately in Bing Webmaster Tools
+and Google Search Console.
+
+Protocol: https://www.indexnow.org/documentation

_config.yml

@@ -192,6 +192,11 @@ defaults:
       path: "googlec07fc31230407367.html"
     values:
       sitemap: false
+  # Ownership proof is a public UTF-8 text file, not a page to index.
+  - scope:
+      path: "aa524f0e6a15332a02207f2fc2b621a9.txt"
+    values:
+      sitemap: false
   # _posts  (essays + study notes — lamp-styled article layout)
   - scope:
       path: ""

aa524f0e6a15332a02207f2fc2b621a9.txt

@@ -0,0 +1 @@
+aa524f0e6a15332a02207f2fc2b621a9

scripts/audit_site.py

@@ -275,6 +275,16 @@ def json_file(name):
     expect("https://floatingcatty.github.io/googlec07fc31230407367.html"
            not in sitemap_urls, "Verification file should not be promoted as an indexable article")
 
+    # The IndexNow key is an ownership proof, not an indexable document.
+    key = "aa524f0e6a15332a02207f2fc2b621a9"
+    keyfile = root / (key + ".txt")
+    expect(keyfile.is_file(), "IndexNow proof missing from deployed site")
+    if keyfile.is_file():
+        expect(keyfile.read_text(encoding="utf-8").strip() == key,
+               "IndexNow proof content mismatches its filename")
+    expect(("https://floatingcatty.github.io/" + key + ".txt") not in sitemap_urls,
+           "IndexNow proof should not appear in sitemap")
+
     research = json_file("research.json")
     papers = json_file("publications.json")
     notes = research.get("items", [])

scripts/indexnow.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""Notify IndexNow only about public URLs changed by the latest site deployment.
+
+For GitHub Pages deployments this runs after a successful deployment_status event.
+To recover from missing deployment events, workflow_dispatch accepts a comma- or
+newline-separated list of site-relative or absolute changed URLs.
+
+It is deliberately independent of any paid API and uses Python's stdlib only.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+import re
+import subprocess
+import sys
+from urllib.parse import urlsplit, urlunsplit
+from urllib.request import Request, urlopen
+from xml.etree import ElementTree
+
+ORIGIN = "https://floatingcatty.github.io"
+HOST = "floatingcatty.github.io"
+INDEXNOW = "https://api.indexnow.org/indexnow"
+PERMALINK = re.compile(r'(?m)^permalink\s*:\s*["\x27]?([^"\x27#\s]+)')
+FRONTMATTER = re.compile(r"\A---\s*\n(.*?)\n---\s*\n", re.S)
+KEYFILE = re.compile(r"[a-f0-9]{32}\.txt\Z")
+BROAD_SOURCE_PREFIXES = ("_includes/", "_layouts/", "_data/", "_sass/", "assets/css/")
+BROAD_SOURCE_FILES = {"_config.yml", "research.json", "publications.json"}
+
+
+def key_at_root(root: Path) -> tuple[str, str]:
+    matches = [
+        p for p in root.iterdir()
+        if p.is_file()
+        and KEYFILE.fullmatch(p.name)
+        and p.read_text(encoding="utf-8").strip() == p.stem
+    ]
+    if len(matches) != 1:
+        raise ValueError("Expected exactly one valid 32-hex IndexNow key file at repository root")
+    key = matches[0].stem
+    return key, f"{ORIGIN}/{key}.txt"
+
+
+def canonical_url(text: str) -> str:
+    text = text.strip()
+    if not text:
+        raise ValueError("URL cannot be empty")
+    if text.startswith("/") and not text.startswith("//"):
+        text = ORIGIN + text
+    parsed = urlsplit(text)
+    if (parsed.scheme != "https" or parsed.netloc != HOST
+            or parsed.username or parsed.password
+            or parsed.query or parsed.fragment):
+        raise ValueError(f"Not a canonical URL on {HOST}: {text}")
+    if not parsed.path.startswith("/"):
+        raise ValueError("Invalid URL path")
+    return urlunsplit(("https", HOST, parsed.path, "", ""))
+
+
+def permalink_from_source(body: str | None) -> str | None:
+    if not body:
+        return None
+    front = FRONTMATTER.search(body)
+    if not front:
+        return None
+    match = PERMALINK.search(front.group(1))
+    return canonical_url(match.group(1)) if match else None
+
+
+def git_text(spec: str, root: Path) -> str | None:
+    proc = subprocess.run(
+        ["git", "show", spec], cwd=root, capture_output=True, text=True, check=False
+    )
+    return proc.stdout if proc.returncode == 0 else None
+
+
+def git_changes(root: Path) -> list[tuple[str, str]]:
+    proc = subprocess.run(
+        ["git", "diff", "--name-status", "-M", "HEAD^", "HEAD"],
+        cwd=root, capture_output=True, text=True, check=False,
+    )
+    if proc.returncode != 0:
+        raise ValueError("Cannot read deployment diff from HEAD^ to HEAD")
+    changes: list[tuple[str, str]] = []
+    for line in proc.stdout.splitlines():
+        fields = line.split("\t")
+        status = fields[0][0]
+        if status in {"R", "C"} and len(fields) >= 3:
+            changes.extend((("D", fields[1]), ("A", fields[2])))
+        elif status in {"A", "M", "D"} and len(fields) >= 2:
+            changes.append((status, fields[1]))
+    return changes
+
+
+def changed_urls(root: Path, changes: list[tuple[str, str]]) -> tuple[set[str], bool]:
+    urls: set[str] = set()
+    all_public_pages = False
+    for status, path in changes:
+        if path in BROAD_SOURCE_FILES or path.startswith(BROAD_SOURCE_PREFIXES):
+            all_public_pages = True
+        elif path == "index.html":
+            urls.add(ORIGIN + "/")
+        elif path.startswith(("_pages/", "_publications/", "_posts/")):
+            # A deleted or relocated permalink must also be reported. Do not
+            # suppress URLs just because the current sitemap omits deleted pages.
+            previous = git_text(f"HEAD^:{path}", root) if status != "A" else None
+            current_path = root / path
+            current = current_path.read_text(encoding="utf-8") if current_path.exists() and status != "D" else None
+            for body in (previous, current):
+                if body is None:
+                    continue
+                permalink = permalink_from_source(body)
+                if permalink:
+                    urls.add(permalink)
+                else:
+                    # Jekyll-generated post permalinks are not reliably inferred
+                    # from filenames without also inspecting the site config.
+                    all_public_pages = True
+    return urls, all_public_pages
+
+
+def sitemap_urls() -> set[str]:
+    with urlopen(ORIGIN + "/sitemap.xml", timeout=20) as response:
+        xml = response.read()
+    root = ElementTree.fromstring(xml)
+    urls = set()
+    for e in root.iter():
+        if e.tag.rsplit("}", 1)[-1] == "loc" and e.text:
+            url = e.text.strip()
+            try:
+                good = canonical_url(url)
+            except ValueError:
+                continue
+            if not urlsplit(good).path.endswith((".xml", ".txt")):
+                urls.add(good)
+    return urls
+
+
+def verify_published_key(key: str, url: str) -> None:
+    with urlopen(url, timeout=20) as response:
+        body = response.read(256).decode("utf-8").strip()
+    if body != key:
+        raise RuntimeError(f"IndexNow key file not yet deployed at {url}")
+
+
+def notify(urls: set[str], key: str, key_url: str) -> int:
+    payload = {
+        "host": HOST,
+        "key": key,
+        "keyLocation": key_url,
+        "urlList": sorted(urls),
+    }
+    data = json.dumps(payload, separators=(",", ":")).encode("utf-8")
+    request = Request(
+        INDEXNOW,
+        data=data,
+        headers={"Content-Type": "application/json; charset=utf-8", "User-Agent": "academic-site-indexnow/1.0"},
+        method="POST",
+    )
+    with urlopen(request, timeout=30) as response:
+        code = response.status
+    if code not in (200, 202):
+        raise RuntimeError(f"IndexNow returned unexpected HTTP {code}")
+    return code
+
+
+def run(root: Path, *, manual_urls: str, dry_run: bool) -> int:
+    key, key_url = key_at_root(root)
+    if manual_urls.strip():
+        raw = re.split(r"[,\n]+", manual_urls)
+        urls = {canonical_url(x) for x in raw if x.strip()}
+        all_pages = False
+    else:
+        urls, all_pages = changed_urls(root, git_changes(root))
+        if all_pages:
+            # One-off legitimate sitewide structural changes (e.g. common
+            # layout updates) affect all rendered pages. Otherwise submit
+            # only the changed explicit permalinks.
+            if not dry_run:
+                urls.update(sitemap_urls())
+            else:
+                print("Dry-run: shared layout changed; sitemap expansion deferred")
+    if not urls and not all_pages:
+        print("No changed public URLs to report")
+        return 0
+    if len(urls) > 10_000:
+        raise ValueError("Too many IndexNow URLs in one submission")
+    print(f"IndexNow URLs: {len(urls)}")
+    for url in sorted(urls):
+        print("  " + url)
+    if dry_run:
+        print("Dry-run: no key verification or network submission")
+        return 0
+    verify_published_key(key, key_url)
+    status = notify(urls, key, key_url)
+    print(f"IndexNow HTTP {status}: notification received (NOT an indexing guarantee)")
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--root", type=Path, default=Path("."))
+    parser.add_argument("--urls", default="", help="Comma/newline-separated canonical or relative changed URLs")
+    parser.add_argument("--dry-run", action="store_true")
+    args = parser.parse_args()
+    try:
+        return run(args.root.resolve(), manual_urls=args.urls, dry_run=args.dry_run)
+    except Exception as exc:
+        print(f"IndexNow setup/submission error: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())