Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ad4628f
feat(sync): negotiate encryption per peer (WIP)
pjwerneck Oct 7, 2026
644ea00
fix(sync): address review findings in encryption negotiation
pjwerneck Oct 7, 2026
b478e76
fix(sync): hold only pending messages and reload policy on event writes
pjwerneck Oct 7, 2026
5e7f55c
fix(sync): keep required peers required and encrypt at rest before wr…
pjwerneck Oct 8, 2026
107b398
fix(sync): harden legacy policy checks, unknown peers and bundle reads
pjwerneck Oct 8, 2026
efc097b
refactor(sync): drop legacy-peer handling and trim encryption code
pjwerneck Oct 8, 2026
e460f59
fix(sync): reload the key file policy on every write at rest
pjwerneck Oct 8, 2026
a9e63e5
fix(sync): strict reads at rest, delta writes for required peers, rej…
pjwerneck Oct 8, 2026
9555080
fix(sync): accept plaintext at rest written before the pass, harden r…
pjwerneck Oct 8, 2026
6de6f37
refactor(sync): simplify encryption negotiation
pjwerneck Oct 8, 2026
967df46
fix(sync): make encryption waits and drops visible
pjwerneck Oct 8, 2026
935f0b9
refactor(sync): address review standards issues
pjwerneck Oct 8, 2026
0e299ca
Merge branch 'dev' into pjwerneck/auto-encryption-settings
pjwerneck Oct 14, 2026
cbc7f7c
fix: update model configuration to forbid extra fields in PeerStore a…
pjwerneck Oct 14, 2026
650e93b
fix: send the held job before the owner syncs; format docs and notebooks
pjwerneck Oct 14, 2026
3ca82ed
test: drop repeated encryption policy tests; fix debug enclave policy…
pjwerneck Oct 14, 2026
6934e4c
Merge remote-tracking branch 'origin/dev' into pjwerneck/auto-encrypt…
pjwerneck Oct 20, 2026
cffcb24
chore: retrigger checks
pjwerneck Oct 20, 2026
6c84133
Merge remote-tracking branch 'origin/dev' into pjwerneck/auto-encrypt…
pjwerneck Oct 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,26 @@ Syft uses Google Drive as its file-based transport layer. All peer-to-peer commu

---

## Encryption Policy

Each client advertises an encryption policy in `SYFT_version.json`:

| Policy | Set with | Effect |
| ---------- | --------------------------- | ----------------------------------------- |
| `optional` | default, `encryption=False` | encrypt only with a peer that requires it |
| `required` | `encryption=True` | encrypt with every peer; refuse plaintext |
| `off` | `encryption="off"` | never encrypt |

A pair encrypts when one side is `required`. An `off` and a `required` client conflict: each side skips the other with a warning.

A client takes its policy at login. The key file records a `required` policy, so later logins (and syft-bg) stay `required`. A process that started with another policy stops at its next sync with an error to restart it. Files at rest written before the switch stay plaintext; new ones are encrypted.

While a peer's policy or keys are not known yet, a change for it waits, with a warning, and a later sync sends it. An owner's change stays pending on disk. A scientist's change waits in memory, and is lost if the process exits.

A pair that encrypts refuses plaintext. The receiver drops a plaintext message with a warning: the owner archives it unread, and a scientist skips it.

A peer whose version file has no policy runs an outdated syft. It is skipped with a warning to upgrade, and a send to it raises.

## Peer Requests

DS creates inbox/outbox folders on their own drive and grants the DO write access. The DO discovers these folders by searching for `syft_datasite#{my_email}#` folders they don't own.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@
"# from google.colab import files; files.upload()\n",
"# CRYPTO_KEYS_PATH = \"crypto_keys.json\"\n",
"\n",
"benchmark_owner = login_do(encryption=True, crypto_keys_path=CRYPTO_KEYS_PATH)\n",
"benchmark_owner = login_do(crypto_keys_path=CRYPTO_KEYS_PATH)\n",
"print(f\" Benchmark owner : {benchmark_owner.email}\")"
]
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
"# from google.colab import files; files.upload()\n",
"# CRYPTO_KEYS_PATH = \"crypto_keys.json\"\n",
"\n",
"model_owner = login_do(encryption=True, crypto_keys_path=CRYPTO_KEYS_PATH)\n",
"model_owner = login_do(crypto_keys_path=CRYPTO_KEYS_PATH)\n",
"print(f\" Model owner : {model_owner.email}\")"
]
},
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

21 changes: 5 additions & 16 deletions packages/enclave-model-api-example/notebooks/demo.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@
"import os\n",
"from pathlib import Path\n",
"\n",
"# Deployed PROD enclave (real Gemma model, encryption ON). Encryption is auto-detected\n",
"# from /model-status in the next cell, so no ENCLAVE_ENCRYPTION env var is needed.\n",
"# Deployed PROD enclave (real Gemma model, encryption ON). The data owners encrypt\n",
"# because the enclave policy requires it, so they need no encryption setting.\n",
"os.environ[\"ENCLAVE_URL\"] = \"http://34.68.46.255:8080\"\n",
"os.environ[\"GEMMA_WEIGHTS_DIR\"] = (\n",
" f\"{Path.home().resolve()}/.cache/kagglehub/models/google/gemma-3/flax/gemma-3-270m-it/1\"\n",
Expand Down Expand Up @@ -65,15 +65,6 @@
"# deployed VM, e.g. \"http://<vm-ip>:8080\" — defaults to a local server if unset.\n",
"ENCLAVE_URL = os.environ.get(\"ENCLAVE_URL\", \"http://localhost:8080\")\n",
"\n",
"# Auto-match the enclave's encryption setting: data owners MUST use the same\n",
"# encryption mode as the enclave or Drive sync silently fails. The enclave\n",
"# reports it via /model-status (\"use_encryption\"); fall back to the\n",
"# ENCLAVE_ENCRYPTION env var if the endpoint doesn't expose it (older image).\n",
"_status = requests.get(f\"{ENCLAVE_URL}/model-status\").json()\n",
"ENCRYPTION = _status.get(\"use_encryption\")\n",
"if ENCRYPTION is None:\n",
" ENCRYPTION = os.environ.get(\"ENCLAVE_ENCRYPTION\", \"false\").lower() == \"true\"\n",
"\n",
"# Local dir holding the real Gemma 3 270m flax weights that DO1 uploads. Expected layout:\n",
"# <WEIGHTS_DIR>/tokenizer.model and <WEIGHTS_DIR>/<checkpoint>/\n",
"WEIGHTS_DIR = Path(\n",
Expand All @@ -82,7 +73,7 @@
" \"~/.cache/kagglehub/models/google/gemma-3/flax/gemma-3-270m-it/1\",\n",
" )\n",
").expanduser()\n",
"ENCLAVE_URL, ENCRYPTION, WEIGHTS_DIR"
"ENCLAVE_URL, WEIGHTS_DIR"
]
},
{
Expand Down Expand Up @@ -111,10 +102,8 @@
"metadata": {},
"outputs": [],
"source": [
"do1 = login_do(DO1, CRED / \"token_do.json\", encryption=ENCRYPTION) # model + log owner\n",
"do2 = login_do(\n",
" DO2, CRED / \"token_ds.json\", encryption=ENCRYPTION\n",
") # submits the analysis job"
"do1 = login_do(DO1, CRED / \"token_do.json\") # model + log owner\n",
"do2 = login_do(DO2, CRED / \"token_ds.json\") # submits the analysis job"
]
},
{
Expand Down
15 changes: 3 additions & 12 deletions packages/enclave-model-api-example/scripts/run_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,6 @@
)
).expanduser()
EXPECT_MODEL = os.environ.get("DEMO_EXPECT_MODEL", "auto").lower() # auto|real|mock
# Encrypt all Drive communication. MUST match the enclave: the prod `inference-start`
# recipe forces SYFT_ENCLAVE_USE_ENCRYPTION=true, so set DEMO_ENCRYPTION=true when
# running against a prod enclave (debug VMs default to encryption off).
ENCRYPTION = os.environ.get("DEMO_ENCRYPTION", "false").lower() in ("1", "true", "yes")

ENCLAVE = "beach.do.008@gmail.com"
DO1 = "koenlennartvanderveen@gmail.com" # model + log owner
Expand Down Expand Up @@ -201,15 +197,10 @@ def main():
"DEMO_EXPECT_MODEL=mock but the enclave is serving the REAL backend."
)
print(f"enclave backend: {'REAL Gemma' if server_real else 'mock'}")
# Match the enclave's encryption setting (reported by /model-status). Fall
# back to DEMO_ENCRYPTION if the endpoint doesn't expose it (older image).
server_enc = status.get("use_encryption")
encryption = ENCRYPTION if server_enc is None else bool(server_enc)

# The data owners encrypt with the enclave when its policy requires it.
banner("log in the two data owners")
print(f"encryption: {encryption}")
do1 = login_do(DO1, CRED / "token_do.json", encryption=encryption)
do2 = login_do(DO2, CRED / "token_ds.json", encryption=encryption)
do1 = login_do(DO1, CRED / "token_do.json")
do2 = login_do(DO2, CRED / "token_ds.json")

banner("peer with the enclave, wait until connected")
do1.add_peer(ENCLAVE)
Expand Down
2 changes: 1 addition & 1 deletion packages/syft-enclave/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ just start-debug EMAIL my-vm n2d-standard-4 # override name / machine type
just stop [name] # Teardown: Deletes the VM.
```

Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`.
Debug enclaves set `use_encryption=false`, so their policy is `optional`: they encrypt only with a peer that requires encryption. Clients select encryption from the enclave policy, so they need no setting.

## GPU deployments

Expand Down
2 changes: 1 addition & 1 deletion packages/syft-enclave/docs/dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ just start-debug # cpu (default)
just hardware=gpu start-debug # gpu (a3-highgpu-1g, 1x H100) — see docs/terraform_cs.md (Confidential Spaces) "GPU deployments"
```

Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`.
Debug enclaves run with encryption off. Clients select encryption from the enclave policy, so they need no setting.

## Building & pushing a new Docker Image

Expand Down
2 changes: 1 addition & 1 deletion packages/syft-enclave/docs/terraform_cs.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Deletes the VM, secret, service account, and IAM bindings. The enabled APIs are

Debug image — SSH enabled, container logs redirected to serial output, encryption off (override with `use_encryption = true` in tfvars), restart policy `Never` (a crashed container stays down for post-mortem; `gcloud compute instances reset` to relaunch).

> Clients must match the enclave's encryption setting: against a debug enclave, data owners log in with `login_do(encryption=False)`.
> Clients select encryption from the policy the enclave advertises, so they need no encryption setting.

```bash
just tf-apply-dev
Expand Down
Loading
Loading