Skip to content

feat: config backup and restore (WI-1721) - #247

Merged
thedancingdeveloper merged 3 commits into
mainfrom
wi-1721-backup-restore
Oct 20, 2026
Merged

thedancingdeveloper merged 3 commits into
mainfrom
wi-1721-backup-restore

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

WI-1721: export and import of settings through the API and the UI, validated on import, with secrets masked. Written by a DeepSeek worker; the handoff is on WI-1721.

🤖 Generated with Claude Code

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator Author

Review of 254635a: 2 blockers. GitHub rust and e2e were still pending when I reviewed; frontend passes.

Checks run locally:

  • cargo test -p nzb-core --lib config_backup passes 11/11, and --test config_backup_api passes 7/7.
  • cargo clippy --workspace --all-targets -D warnings and fmt --check are clean.
  • ng test passes 290/290 and lint is clean.

Blockers

  1. The "safe to share" export leaks indexer API keys through rss_feeds[].url. The module doc and ?secrets default promise that a masked export "can be shared, committed or attached to an issue without exposing a credential". But mask_secrets never touches RSS feed URLs, and indexer RSS URLs carry the user's key in the query (…/rss?t=2000&apikey=…, …&r=…&i=…). The masked export therefore publishes every indexer credential. The canary tests (assert_no_secret) seed no RSS URL, which is why this passed. Mask the query string, or at least known key parameters (apikey, api_key, r, i, key, token, passkey) plus any user:pass@ userinfo. Make merge_secrets keep the stored URL when the masked form matches, the way the proxy URL already works. Add an RSS canary to seeded() / assert_no_secret. While there:
    • notifications.ntfy_topic is effectively a bearer secret on public ntfy servers (anyone with the topic name can read and post); mask it too.
    • Treat otel.*endpoint userinfo like the proxy URL.
  2. The lockfile churn is back. Cargo.lock downgrades windows-sys 0.61.2 → 0.60.2 in 10 places, the same regeneration feat(par2): in-flight repair fold state (WI-1235) #240, feat(par2): drive the repair fold from the download engine (WI-1236) #242 and feat(par2): post-processing solves from the in-flight fold (WI-1237) #244 each had to drop. Only the nzb-core 0.2.21 → 0.2.22 line should change. The bump itself is right: config_backup is new public API.

Import validation: sound

  • Size and encoding. Bodies over 4 MiB get a 413, and non-UTF-8 gets a 400.
  • Versions. backup_version newer than 1 is refused, and so is a config_version newer than this build.
  • Unknown keys. Inside [config] they are refused, because AppConfig and every section are deny_unknown_fields.
  • Validation. Normalization matches AppConfig::read (server ids, retention, RSS expiry, migrate). Then the full AppConfig::validate, each CategoryConfig::validate, and a non-empty category list are checked. Any problem is a 400 listing every reason, and nothing is written.
  • Preview. It takes the identical path minus the commit.
  • Masks. They are keep tokens, matched by server id: kept when this install has the value, cleared with a warning when it doesn't, never written through as a literal ********.
  • Apply and auth. The restore is one update_config_with, then a live re-apply through the same setters the settings routes use. Start-up-only keys are listed in restart_required. All three routes require auth (tested), and the export is no-store.

Nits

  • The envelope accepts anything. BackupDocument and BackupMeta aren't deny_unknown_fields, and BackupMeta::default() sets backup_version = 1. So a [backup] table without backup_version parses as version 1, the version == 0 "not a rustnzb backup" check can never fire, and typos in the header are silently ignored. Default the version to 0 in serde, or require it.
  • A masked proxy URL can be written through literally. If its host or port changed in the backup, socks5://********@newhost no longer equals mask_proxy_userinfo(stored), so it's saved as-is with ******** as credentials. Warn or clear, the same as for an unknown server.
  • TOML parse errors can echo a secret. not a valid rustnzb backup: {e} includes the offending source line, so a syntax error near a password in a secrets-included file echoes that line in the 400 response. It only goes back to the uploader, but don't log it.

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator Author

Re-review of d1d7c78: no blockers (CI pending). GitHub rust, frontend and e2e were still pending when I reviewed; CodeQL passes.

Both blockers are fixed:

  1. Indexer keys and other secrets are no longer in a masked export.
    • rss_feeds[].url goes through mask_url_secrets, which masks the userinfo and the whole query string and keeps the scheme, host, path and fragment, so the file still identifies the feed.
    • notifications.ntfy_topic is masked and merged like the other notification fields.
    • The otel.*endpoint userinfo is masked.
    • The canaries now cover an RSS key, an ntfy topic and an OTel secret, in both config_backup unit tests and the API test (a_masked_export_masks_indexer_keys_and_ntfy_topics).
  2. The lockfile is clean. Against main, Cargo.lock changes only nzb-core 0.2.21 → 0.2.22; the windows-sys churn is gone.

The restore merge is sound.

  • RSS: a masked feed keeps this install's URL only when both the name and the masked form match. Otherwise the feed is dropped with an explicit warning, never saved with ?********.
  • OTel: a masked endpoint is restored when it matches, and otherwise reset to the default or unset, with a warning.
  • Proxy URLs: a masked proxy URL that doesn't match is now cleared with a warning, which was a nit last time.
  • Final check: check_masks runs after the merge as a backstop. Any surviving ******** in a password, key, notification field, proxy, RSS URL or OTel endpoint makes it a 400, and nothing is written.

The other two nits from my first review are also fixed:

  • BackupDocument and BackupMeta are now deny_unknown_fields, and backup_version defaults to 0, so a header without it is refused.
  • Parse errors report line and column without echoing the source line.

Checks run locally:

  • cargo test -p nzb-core --lib config_backup passes 17/17, and --test config_backup_api passes 9/9.
  • cargo clippy --workspace --all-targets -D warnings and fmt --check are clean.

Nits

  • Path-embedded keys still leak. mask_url_secrets covers query-string keys, which is what newznab and most indexers use, but a key embedded in the path survives (for example https://indexer/rss/<uid>/<apikey>, a style some private trackers use). Mention this in docs/CONFIGURATION.md's backup section so nobody assumes every feed URL is scrubbed, or offer a "mask whole URL" option.
  • A masked ntfy_server with userinfo isn't masked. That's uncommon, but mask_proxy_userinfo would cover it for free.

Add GET /api/config/backup, POST /api/config/backup/preview and
POST /api/config/backup/restore, plus a Backup & restore page in
Settings.

A backup is a `[backup]` header over the `[config]` table, so it is a
readable config.toml with a provenance envelope. The default export
masks every stored secret as `********` and says
`secrets_included = false`; `?secrets=true` is an explicit opt-in that
writes the real values and labels the file.

On restore the mask is a keep token: a masked secret keeps the value
the target already holds, and a mask with nothing behind it leaves the
setting unset and is reported as a warning rather than written through
as a literal password. A secret absent from the backup is cleared,
because a restore replaces the file layer.

Nothing is written unless the document parses, normalizes and
validates; unknown keys, newer envelope or config versions and invalid
values are refused with the reason and leave the config and the file
untouched. The preview takes the same path up to the commit, so the UI
can show the summary, kept secrets, warnings and restart-required
settings before the operator confirms. Live settings are pushed into
the running queue manager after the save.

Move the shared secret masking out of the handlers into
`nzb_core::config_backup` so the settings endpoint and the backup
scrub exactly the same fields. Bump nzb-core to 0.2.22 for the new
public module.

Tests: unit coverage for masking, merge and validation; API tests for
the no-secret export, keep-on-mask, replace-with-secrets, fresh-install
warnings, refusals leaving state untouched and auth; frontend tests
for the download, preview gating, confirmation and failure paths.
A masked export promised to be safe to share, but it left RSS feed URLs
untouched. Indexers carry the account key in the URL query, so every
masked backup published each indexer credential. Mask where a secret can
actually live:

- RSS feed URLs: the query string and any userinfo are replaced, keeping
  scheme, host and path so a restore can still match the feed. A masked
  URL keeps the stored one only for the same feed; with nothing to keep
  the feed is dropped with a warning rather than saved with `********`.
- notifications.ntfy_topic, which is a bearer secret on a public ntfy
  server.
- otel.endpoint, logs_endpoint and metrics_endpoint userinfo.

After the merge, validate() refuses any remaining mask, so a masked
proxy URL pointing at a different host is cleared with a warning instead
of being written through literally.

Also tighten the envelope: `backup_version` now defaults to 0 rather than
1, so a document that omits it is refused instead of read as current, and
the header and top-level tables reject unknown keys. A TOML syntax error
is reported with line and column but without echoing the offending source
line, which could otherwise copy a password out of a secrets-included
file into the 400 response.

Drop the regenerated windows-sys downgrades from Cargo.lock; the nzb-core
0.2.22 line is the only change.

Tests: RSS, ntfy-topic and OTel canaries in the core and API suites;
keep-on-match and drop-without-match feed restores; masked-proxy mismatch;
envelope version/key refusals; parse-error redaction.
CodeQL flags a test panic message that interpolates the canary it is
checking for as cleartext logging of sensitive information. Report which
credential leaked by field name instead; the assertions stay as strict.
@thedancingdeveloper
thedancingdeveloper added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 8f9874c Oct 20, 2026
12 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the wi-1721-backup-restore branch October 20, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants