[jatinbansal1998]

Why

Bulk-labelling a few thousand Gmail messages (search, then pipe the IDs into gmail batch modify) kept failing with:

Google API error (403 rateLimitExceeded): Quota exceeded for quota metric 'Total Query Cost' and limit 'Units per minute per user' of service 'gmail.googleapis.com' ...

Two causes:

1. gmail messages search and gmail search fetch headers and labels for every result: one extra API call per result. A job that only needs IDs pays for 2,000+ detail calls. --select trims the output after the fetch, so it does not help.
2. The 403 rate-limit retry added in 60ae623 gives up after three backoffs (1s, 2s, 4s plus jitter, 7–10.5s in total). Gmail's quota is per minute, so the retries run out before it refills.

What changed

• --ids-only on gmail messages search (id, threadId) and gmail search (thread id). It returns what the list call already holds and skips the label lookup and the per-result fetch. Works with --all, --max, --page, --count, --json, --plain. It rejects the content flags it cannot serve (--include-body, --full, --include-attachments, --include-recipients, --headers, --sanitize-content, --oldest).
• MaxRateLimitRetries goes from 3 to 6. Six backoffs wait 63s to just under 94.5s in total, which covers one minute window. This is in the shared RetryTransport, so every Google API command gets it. The backoff code (jitter, Retry-After handling, the 403-reason check) is unchanged.
• Docs: docs/gmail-workflows.md (bulk-ID recipe), docs/automation.md (retry budget), regenerated docs/commands/*, CHANGELOG.

Before and after

Input: gog --readonly gmail messages search 'list:substack.com' --all --max 500 --json, run against a real Gmail account with 2,172 matches. HTTP attempts were counted with a throwaway counter hooked into RetryTransport.RoundTrip. It was only in the measurement builds, not in this PR.

Before (upstream main f9052a5), run twice, both failed:

before-main: exit=7 time=25.3s http_attempts=394
      1 200 GET .../users/me/labels
      5 200 GET .../users/me/messages
    348 200 GET .../users/me/messages/{id}
     38 403 GET .../users/me/messages/{id}
      2 0   GET .../users/me/messages/{id}      (cancelled after the failure)
Google API error (403 rateLimitExceeded): Quota exceeded for quota metric 'Total Query Cost' and limit 'Units per minute per user' ...

After, same command (retry budget 6): finishes, slowly, because the quota caps it:

after-full-retry6: exit=0 time=346.3s http_attempts=2886
      1 200 GET .../users/me/labels
      5 200 GET .../users/me/messages
   2172 200 GET .../users/me/messages/{id}
    708 403 GET .../users/me/messages/{id}      (rate limits absorbed by backoff)

After, with --ids-only:

after-idsonly:       exit=0 time=3.2s http_attempts=5   (5 x 200 GET .../users/me/messages)
after-idsonly-plain: exit=0 time=2.8s http_attempts=5   (--plain: header "ID<TAB>THREAD" + 2,172 rows)
threads-idsonly:     exit=0 time=3.4s http_attempts=5   (gog gmail search ... --ids-only: 2,169 threads, keys ["id"])
page-idsonly:        exit=0 time=1.3s http_attempts=1   (--max 2 --page <token>: count 2, has_more true)

The --ids-only run returned the same 2,172 id+threadId pairs as the full run (sorted diff empty).

Upgrade and fresh-install proof

Same machine, same existing config and file keyring: installed v0.46.0, then this PR's build. Then this PR's build with GOG_HOME pointed at an empty temp directory and no keyring password. Settings and the stored account are unchanged across the upgrade. The fresh install starts cleanly, reports no accounts, fails with the usual missing-credentials error (exit 10) instead of crashing, and rejects bad --ids-only combinations with a usage error (exit 2). Account email redacted.

Output

### v0.46.0 (e0e3cb9c 2026-10-21T05:43:32Z)
$ gog config path
/home/dell/.config/gogcli/config.json
exit=0
$ gog config list --json
{
  "gmail_no_send": "false",
  "keyring_backend": "file",
  "path": "/home/dell/.config/gogcli/config.json",
  "places_api_key": "",
  "timezone": "",
  "youtube_api_key": ""
}
exit=0
$ gog auth list --plain
<account>	default	gmail	2026-10-22T19:00:38Z	oauth
exit=0
$ gog --readonly --account <account> gmail messages search 'list:substack.com' --max 3 --json --results-only --select id
[
  {
    "id": "1a167a481aa596ff"
  },
  {
    "id": "1a16440b9ab34592"
  },
  {
    "id": "1a1643fc2f15d497"
  }
]
exit=0
### v0.46.0-9-g3be88741 (3be88741cc37 2026-10-23T10:27:39Z)
$ gog config path
/home/dell/.config/gogcli/config.json
exit=0
$ gog config list --json
{
  "gmail_no_send": "false",
  "keyring_backend": "file",
  "path": "/home/dell/.config/gogcli/config.json",
  "places_api_key": "",
  "timezone": "",
  "youtube_api_key": ""
}
exit=0
$ gog auth list --plain
<account>	default	gmail	2026-10-22T19:00:38Z	oauth
exit=0
$ gog --readonly --account <account> gmail messages search 'list:substack.com' --max 3 --json --results-only --select id
[
  {
    "id": "1a167a481aa596ff"
  },
  {
    "id": "1a16440b9ab34592"
  },
  {
    "id": "1a1643fc2f15d497"
  }
]
exit=0
### fresh install: v0.46.0-9-g3be88741 (3be88741cc37 2026-10-23T10:27:39Z), GOG_HOME=<empty temp dir>
$ gog config list --json
{
  "gmail_no_send": "false",
  "keyring_backend": "",
  "path": "<tmp>/config/config.json",
  "places_api_key": "",
  "timezone": "",
  "youtube_api_key": ""
}
exit=0
$ gog auth list --plain
No tokens stored
exit=0
$ gog --readonly --account <account> gmail messages search 'list:substack.com' --ids-only --json
OAuth client credentials missing (OAuth client ID JSON).
Download from: https://console.cloud.google.com/apis/credentials (Create Credentials → OAuth client ID → Desktop app → Download JSON)
Then run: gog auth credentials <credentials.json> (expected at <tmp>/data/credentials.json)
{"error":{"class":"config","code":10,"message":"OAuth client credentials missing (OAuth client ID JSON).\nDownload from: https://console.cloud.google.com/apis/credentials (Create Credentials → OAuth client ID → Desktop app → Download JSON)\nThen run: gog auth credentials \u003ccredentials.json\u003e (expected at <tmp>/data/credentials.json)"}}
exit=10
$ gog gmail messages search x --ids-only --include-body
--ids-only cannot be used with --include-body, --full, --include-attachments, --include-recipients, or --sanitize-content
exit=2
files created in GOG_HOME:
./data/keyring/.lock

Rebased on 0.47.1

Rebased onto b8298ae6 (after the 0.47.1 release); the CHANGELOG entries are under 0.47.2 - Unreleased. 0.47.0 added --headers to gmail messages search (#1249). That flag needs the per-message fetch, so --ids-only now rejects it, with a test. Live re-check on the rebased build (--readonly):

messages search 'list:substack.com' --all --max 500 --ids-only: exit=0 time=6.3s count=2176 keys=["id","threadId"]
gmail search 'list:substack.com' --max 2 --ids-only:            {"n":2,"keys":["id"]}
messages search ... --max 1 --headers List-Id (no --ids-only):  {"headerKeys":["List-Id"]}
messages search x --ids-only --headers List-Id:                 exit=2 "--ids-only cannot be used with ... --headers ..."

make ci passes on the rebased head.

Testing

• make ci passes (fmt-check, golangci-lint, deadcode, go test, docs-check).
• New tests, each seen failing with the code broken and passing once restored:
  • TestExecute_GmailMessagesSearch_IDsOnlySkipsMessageFetches: --all --ids-only over two pages makes only list calls and returns IDs without metadata.
  • TestExecute_GmailMessagesSearch_IDsOnlyPlainPrintsIDColumns: plain output is exactly ID\tTHREAD rows.
  • TestExecute_GmailMessagesSearch_IDsOnlyRejectsContentFlags: --ids-only --include-body is a usage error.
  • TestExecute_GmailSearch_IDsOnlySkipsThreadFetches: same check for thread search.
  • TestRetryTransport_RateLimit403_DefaultBudgetOutlastsPerMinuteQuota: with the default budget, six 403 rateLimitExceeded responses in a row still succeed, and the total backoff is at least 1 minute. With the old value 3 it fails: total backoff = 8.12s over 3 retries, want at least 1m.
• The live runs above used --readonly. The documented pipeline was also checked live, as a dry run with no changes made: gmail messages search ... --ids-only --json --results-only | jq -r '.[].id' | xargs -n 1000 gog --dry-run gmail batch modify --add-label Newsletters printed a gmail.batch.modify dry-run request with the 3 IDs.
• An independent model review (Grok 4.7) led to the third commit: the retry wait range is now stated exactly, the docs pipeline example now works, the help text is tightened, and a test covers --ids-only with --oldest.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge.

What this changes

This PR adds ID-only Gmail searches and increases the shared Google API rate-limit retry budget from three to six retries.

Example: Search a Gmail account with 2,172 messages matching list:substack.com using --all --max 500.

• Before: The supplied main-branch trace fetches message details and fails with 403 rateLimitExceeded.
• After: The supplied --ids-only trace returns the same 2,172 message and thread ID pairs through five list requests.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The focused patch has strong shipped-CLI evidence and no identified correctness finding, but overall readiness remains capped by the unresolved product decision.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): Real-account CLI traces exercise both search commands through Gmail and RetryTransport, showing ID-only pagination and output plus successful recovery from repeated quota errors; current-head fresh-install and v0.46.0 upgrade output resolves the earlier validation request.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Product

Kind: Feature · Worth it: Needs a maintainer decision · Fix scope: Complete
User problem: Bulk Gmail jobs fetch metadata they do not need and can exhaust retry attempts before a per-minute quota recovers.
Reason: The live results establish clear value, while adding a CLI option and changing an owner-built service-wide default require recorded owner direction.

Merge readiness

⛔ Blocked before merge - 1 item remains

Keep open: this PR provides useful behavior absent from current main, resolves the earlier changelog finding, and supplies the requested upgrade evidence; the new flag and shared retry default still need an owner direction decision.

Priority: P2
Reviewed head: 3be88741cc37f152795b8099abbcd61741ca5aa7
Owner decision: Required. See Decision needed.

Decision needed

• Question: Should Gmail searches expose --ids-only, and should all shared Google API clients use six rate-limit retries?
• Recommendation: Accept both changes: Approve the opt-in ID-only interface and longer shared retry budget, retaining the existing cancellation and Retry-After behavior.
• Why: The measured benefit is substantial, but no maintainer approval of this new CLI option and changed global default appears in the discussion or reviews.

Before merge

• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

None.

Agent review details

How this fits together

Gmail search commands turn queries into paginated Google API requests and format results as JSON or tables; the shared RetryTransport handles rate-limit recovery for Google API clients.

flowchart TD
  A[Gmail search command] --> B[Paginated Gmail list requests]
  B --> C[ID only projection]
  B --> D[Label and detail requests]
  C --> E[JSON or table output]
  D --> E
  B --> F[Shared retry transport]
  D --> F

Loading

Technical review

Best possible solution:

Approve the ID-only search interface and six-retry shared budget if the owner accepts the longer rate-limit recovery window.

Do we have a high-confidence way to reproduce the issue?

Current-main source unconditionally fetches search details and permits only three rate-limit retries, matching the contributor's real-account failure trace; no target code was executed during this review.

Is this the best way to solve the issue?

The patch uses existing search and transport owners, preserves normal search output, and directly removes unnecessary detail requests; broadening the shared retry default remains an owner policy choice.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against f9052a583e87.

Provenance checked

• Gmail search metadata fetching and presentation changes intended behavior with a stated reason (feat(gmail): add message-level search + body decoding #88: Message search was designed to return one metadata-rich row per message; subsequent presentation and recipient changes retained that default.)
• Shared rate-limit retry budget changes intended behavior with a stated reason (feat: add retry logic, concurrent fetching, and extended commands #18: The original transport added bounded exponential retries for resilience, and fix: correct Gmail retries, queries, and Drive output #1253 reused that budget for short-term 403 limits; the PR explains why three retries are too short for the measured Gmail quota window.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

• Introduced change and current-main boundary: The pinned diff adds both ID-only branches and changes the retry constant from three to six; current main still fetches labels and details before formatting search output. (internal/cmd/gmail_messages.go:124, 3be88741cc37)
• Existing capability does not expose cheap ID searches: The existing searchMessageIDs helper serves bulk mutation commands, while --select transforms already-fetched JSON; neither supplies the proposed search output without detail requests. (internal/cmd/gmail_archive.go:256, 3be88741cc37)
• Live behavior and upgrade proof: The complete PR body supplies real-account CLI traces for message/thread ID searches, pagination, plain output, matching ID sets, and quota recovery; current-head output also shows retained v0.46.0 settings and account access plus clean-home handling. Captured body sourceRevision: 0e5bddcf8fcd568d342fba138d2709e5c82c9121dffa64bf98a9fb7460a8de12. (3be88741cc37)
• Earlier finding resolved: Both Unreleased changelog entries now include the PR reference and contributor thanks, satisfying the root AGENTS.md rule and the earlier review finding. (CHANGELOG.md:8, 3be88741cc37)
• Prior retry intent: The merged fix: correct Gmail retries, queries, and Drive output #1253 deliberately added short-term 403 rate-limit handling using the existing bounded shared budget; it does not approve this PR's longer global default. (internal/googleapi/transport.go:167, 60ae623da5b5)
• Related work has a different boundary: gmail batch modify: support --query/--max (cheap bulk relabel by query) #1199 requests query-driven batch modification, while fix(gmail): chunk batch modify requests at 1000 message IDs #1209 handles oversized mutation requests; neither owns reusable ID-only search output. (internal/cmd/gmail_batch.go:70, f9052a583e87)

Likely related people:

• Peter Steinberger: Raw commit bbb084c adds internal/cmd/gmail_presentation.go:12 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: bbb084ca8296; files: internal/cmd/gmail_presentation.go)
• Mariano Belinky: Raw commit c340914 adds internal/cmd/gmail_messages.go:106 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: c340914688f7; files: internal/cmd/gmail_messages.go)

Review metrics

Metric	Value	Why it matters
Production and test changes	Production +98/-30 lines; tests +170/-0 lines	The implementation reuses existing pagination, item types, formatting, and retry ownership; command-path tests cover the distinct new output and validation behavior.

Labels

Label changes:

• add proof: sufficient: Contributor real behavior proof is sufficient.
• add status: ⏳ waiting on maintainer: Only explicit maintainer approvals or decisions remain; merge is still blocked.
• remove status: 📣 needs proof: Current PR status label is status: ⏳ waiting on maintainer.

Label justifications:

• P2: This improves bulk Gmail workflows with substantial request savings and quota recovery, without evidence of an urgent general outage.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: ⏳ waiting on maintainer: Only explicit maintainer approvals or decisions remain; merge is still blocked.
• proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)

• reviewed 2026-10-23T09:05:29.254Z sha 6569c32 :: needs real behavior proof before merge. :: [P3] Add the required changelog reference and contributor credit

Reviewed October 23, 2026, 6:31 AM ET / 10:31 UTC (Revision 2).

[jatinbansal1998]

Added upgrade (v0.46.0 → head) and fresh-install proof to the PR body, and the #1257 reference plus contributor credit to both CHANGELOG entries.

@clawsweeper re-review

[clawsweeper]

🦞👀
Exact review queued.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/39939924162
• Updated: 2026-10-23T10:33:34.427Z

[steipete]

Accepting both the opt-in ID-only search and the longer rate-limit retry budget. The new flag avoids unnecessary detail requests while preserving paging and output envelopes; the retry change keeps cancellation and Retry-After handling intact.

Reproduced the missing flag and short retry window on main, then passed the focused ID-only/RetryTransport tests and full make ci on the proposed revision. govulncheck ./... found no reachable vulnerabilities, and independent review found no actionable P0–P2 issues.

[steipete]

Landed as 6ad392f. Thank you @jatinbansal1998 for the implementation and live-account evidence.

Verified the missing flag and sub-minute retry budget on the previous main, then passed go test ./internal/cmd ./internal/googleapi -run 'IDsOnly|RetryTransport' -count=1, full make ci on Crabbox, and govulncheck ./... with no reachable vulnerabilities. Independent review reported no actionable P0–P2 findings. Exact-head CI passed Linux, minimum Go, macOS, Windows, and worker checks; the Docker build also passed.

[jatinbansal1998]

Thank you @steipete