Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b02def0
refactor(connect_tcp): add transfer control parsing and identity foun…
F18-Ray Nov 16, 2026
01de684
refactor(connect_tcp): add sender-authoritative transfer control state
F18-Ray Nov 16, 2026
fd189d1
feat(connect_tcp)!: integrate typed bidirectional transfer control
F18-Ray Nov 16, 2026
86223f1
fix(tcp): isolate relay control and preserve upstream pause intent
F18-Ray Nov 16, 2026
11ba6ab
fix(connect_tcp)!: gate ordinary resume on per-object ownership
F18-Ray Nov 16, 2026
25aac75
fix(connect_tcp)!: gate relay recovery on shared source ownership
F18-Ray Nov 16, 2026
a7ffd79
Docs: updating docs
F18-Ray Nov 16, 2026
5f7357b
test(connect_tcp): fix cross-version and Windows transfer control fix…
F18-Ray Nov 17, 2026
d4cb4ab
test(connect_tcp): preserve source identity in relay reconnect scenarios
F18-Ray Nov 17, 2026
3b438af
fix(connect_tcp)!: bind transfer cleanup to control sessions
F18-Ray Nov 17, 2026
d1900e7
fix(connect_tcp): arbitrate relay-wide pause and start at the source
F18-Ray Nov 18, 2026
1e4b622
fix(tcp)!: restore numeric resume requests and parallelize target rec…
F18-Ray Nov 18, 2026
12b0ccc
fix(tcp): reconfirm completed relay uploads at EOF
F18-Ray Nov 19, 2026
b0e2783
fix(tcp): reject native file forwarding to self
F18-Ray Nov 19, 2026
8b389f0
docs(transfer): close C6 lifecycle acceptance evidence
F18-Ray Nov 19, 2026
23cbc57
fix(tcp): reserve the receive FID before publishing a relay route
F18-Ray Nov 21, 2026
c161ae4
fix(tcp): require framed writes for resume controls
F18-Ray Nov 21, 2026
dfe0c4f
fix(tcp): reject invalid resume formatter offsets
F18-Ray Nov 21, 2026
b2889b8
test(tcp): the test port unchanged when the resume connect completed
F18-Ray Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ jobs:
os: [ubuntu-latest, windows-latest]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
runs-on: ${{ matrix.os }}
timeout-minutes: 10
timeout-minutes: 30
steps:
- name: Checkout code
uses: actions/checkout@v7
Expand Down
29 changes: 29 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# 文件转发

本词汇表区分转发任务的交付结果与单条数据流的传输位置。

## Language

**转发任务**:将一个文件或文件夹条目交付给一组已接受目标的整体工作;单个目标完成不等于整个任务完成。
_Avoid_:将单条数据流称为整个任务

**任务交付完成**:转发任务的所有已接受目标均已真实确认收到完整文件的状态。
_Avoid_:发送完毕、单目标完成、仅凭文件大小认定成功

**零正文恢复**:恢复位置等于文件总大小、没有剩余文件正文需要发送的恢复;它本身不是任务交付完成的证据。
_Avoid_:把 EOF 等同于全部目标成功

**最终确认**:向源端确认整个转发任务已交付完成的确认,而非只确认某条数据流结束。
_Avoid_:把单流结束称为整任务成功

**补最终确认**:任务已经交付完成、源端尚未获得最终确认时,再次确认原任务完成的过程。
_Avoid_:重新交付、重新传输正文

**转发任务路由**:关联任务源端与各目标接收身份的任务关系;任务已交付但尚未最终确认时仍可保留。
_Avoid_:磁盘文件路径、另一份字节进度

**任务上传进度**:源端为整个转发任务保留的文件发送与恢复依据;它不是某一个目标的独立收件进度。
_Avoid_:仅凭记录中的一个目标身份认定为该目标专属任务

**目标完成集合**:原转发任务中已经真实确认收到完整文件的已接受目标集合。
_Avoid_:已发送字节、零正文恢复请求
2,053 changes: 1,329 additions & 724 deletions PyFlow/network_api/connect_tcp.py

Large diffs are not rendered by default.

337 changes: 337 additions & 0 deletions docs/plans/transfer-control-lifecycle-plan.md

Large diffs are not rendered by default.

595 changes: 595 additions & 0 deletions docs/plans/transfer-control-lifecycle-report.md

Large diffs are not rendered by default.

121 changes: 121 additions & 0 deletions docs/plans/transfer-control-lifecycle-requirements.md

Large diffs are not rendered by default.

225 changes: 225 additions & 0 deletions docs/plans/transfer-pause-start-direction-plan.md

Large diffs are not rendered by default.

594 changes: 594 additions & 0 deletions docs/plans/transfer-pause-start-direction-report.md

Large diffs are not rendered by default.

235 changes: 176 additions & 59 deletions docs/source/File_Transfer/File_Transfer.rst

Large diffs are not rendered by default.

111 changes: 93 additions & 18 deletions docs/source/Network_APIs/TCP_Client_APIs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -297,15 +297,61 @@ commands sent by the server, such as:
``/crypto_key_exchange_ack``, ``/crypto_reject``,
``/crypto_pub_key``, ``/crypto_ready``: the encrypted-channel
handshake messages; a ``/crypto_mode`` mismatch closes the connection.
- ``/pause_trans``, ``/start_trans`` and ``/forward_error <fid> <reason>``:
transfer control and FID-scoped forwarding failures. Forward uploads use
the common ``/server_file_transfer_port`` reply; no ``/forward_upload`` line.
- ``/pause_trans <fid> <type> <issuer_role>``,
``/start_trans <fid> <type> <issuer_role>`` and
``/forward_error <fid> <reason>``: typed transfer control and FID-scoped
forwarding failures. The control lines require exactly four tokens. Type is
``0`` for ordinary transfers or ``1`` for relay legs; issuer role is ``0``
for the data sender or ``1`` for the data receiver. Forward uploads use the
common ``/server_file_transfer_port`` reply; no ``/forward_upload`` line.
- ``/resume_trans <fid> <offset> <origin> <target>``: the sole resume command,
with shell-quoted tuple endpoints. Only an origin matching this control
connection's own identity may resume a local upload. The existing upload row
supplies the source path; a non-server target is replayed through the server
with its stable upstream fid. The receiver's retained full path handles folder
items without copying directory or destination metadata at the source.
Public ``resume_trans`` calls return ``True`` for a newly reserved worker;
busy and duplicate calls return ``False`` without closing the owner. Use
``start_trans`` for a paused live connection. Receiver requests wait for old
receive cleanup, including ``finally`` and file closure, then read the actual
disk size. Offsets are nonnegative integers only; there is no ``?`` query or
``None`` parser/formatter offset.
For forwarded recovery, Server first waits for every old owner of the
original route to finish naturally, without forced teardown. It then directs
independent per-Target Source-to-Server-to-Target streams using each actual
offset. Each has its own progress, resources and failure handling; one body
or final ACK cannot block another Target, including a late arrival. Only
port negotiation on the same actual Server socket plus fid is briefly
serialized, not body transfer or final ACK. Workers retain their admitted
control socket; stable fids, paths and persistence schema do not change.
Source arbitrates global pause/start for all live task owners; the final
owner releases shared control state. A real Target ACK marks only that
Target; no intermediate stream fabricates a whole-route ACK or deletes the
Source task row. All accepted Targets must really acknowledge for task
success. Source disconnection leaves requests queued at Server; Target
disconnection discards its old-socket request and requires a new actual disk
offset after reconnect. There is no automatic retry, and one Target's failure
does not cancel siblings. Strict same-session/fid generation isolation remains
deferred; C5 self-target rejection and C6 combined acceptance are later stages.

After all Targets really ACK, a ``False``/exception writing the upstream ACK
retains the Server route and Source's shared upload row. For C4, Source calls
``resume_trans(original_target, upstream_fid, total)`` explicitly. ``True`` means
worker registration, not completion; a live owner, including ``finally`` cleanup,
returns busy without closing or queueing, so repeat explicitly after it exits.
Only Source connects to Server, exchanging the existing port reply, complete
filename/size header and final ACK with zero body bytes. Offset must equal the
size in this attempt's header exactly. No Target leg or body replay is created;
the original Target may be offline. Server uses its existing
``originator``/``targets``/``completed`` route, not historical name/size/hash
validation or a new byte-progress/terminal-state table. False/exception retains
state; successful ACK write removes only the owning route, and Source removes
its shared row only when it observes that ACK. Reloading existing snapshots
still requires this explicit action, without automatic retry, TTL or schema/wire
changes. A successful Server ACK write lost before Source receives it remains
ambiguous. Ordinary transfers, C3 and global pause/start are unchanged. The
control-layer extension leaves Source ``file_transfer_mode`` and ``_send_file``
unchanged; see :ref:`transfer-pause-and-reconnect`.

For a new relay upload, the source client allocates the upstream FID before it
sends ``/forward_item ... fid=<fid>``. The server returns that same upstream FID
Expand Down Expand Up @@ -629,9 +675,18 @@ Built-in client console commands (user-typed) include:
client to server, preserving the directory structure.
- ``/multiple_file_folder <folder1> <folder2> ...``: sends multiple
folders from client to server.
- ``/forward_file``, ``/forward_folder``, ``/forward_send_msg``: the
forwarding extension's console commands (client-only; the server refuses
them).
- ``/forward_file``, ``/forward_folder``: native Client-only streamed forwarding.
Their shared per-file entry refuses the Client's own complete control
endpoint before creating an upload row, allocating a FID or sending a relay
request. Mixed lists skip only self; same IP/different port and different
IP/same port remain valid. The actual control socket identifies self; a
configured host alias must resolve to that IP (wildcard aliases its actual
port). A configured IP not used after no bind or bind fallback is not self.
Web file/folder forwarding and retained-row replay use the same check;
rejected replay keeps the existing row, and a resume boolean remains worker
registration rather than success. Server console use is refused.
- ``/forward_send_msg``: native Client-only message forwarding; its existing
target rules are unchanged by the file/folder self-target guard.

Ctrl-C and EOF close the connection, send a final ``/quit`` and wait
0.5 seconds before the process ends.
Expand Down Expand Up @@ -840,13 +895,33 @@ And the low-level send mode function is:

def file_transfer_mode(
self, filename, server_address, server_port, client_port,
control_socket=None, crypto=None, fid=None, peer=None, resume_offset=0)

``((host, port), fid)`` keys the leg in ``file_upload_progress``. The
``/pause_trans`` / ``/start_trans`` lines and local methods address that server
scope; an explicit local ``peer`` must be the server, not the logical target.
The value's ``peer`` is the original receiver; ``resume_offset`` continues
the interrupted file from the byte the receiver already holds.
control_socket=None, crypto=None, fid=None, peer=None, resume_offset=0,
trans_type=0)

``((host, port), fid)`` keys the leg in ``file_upload_progress``. Ordinary
senders use ``trans_type=0``; forwarding callers pass ``trans_type=1``.
``pause_trans(fid, peer=None, *, trans_type, direction)`` and
``start_trans(fid, peer=None, *, trans_type, direction)`` require both keyword
arguments. ``direction`` describes the local data role: ``0`` sender or ``1``
receiver. These methods apply local control and notify the server. An omitted
``peer`` selects the current server; an explicit peer may use its configured
address or concrete control endpoint, not the logical forwarded target. Control
identity is canonicalized to ``((client.host, client.port), fid, trans_type,
local_role)`` independently of the progress row's logical receiver.

The ordinary sender, or the relay Source, owns confirmed pause state. Target
requests travel through Server to Source and affect the whole route; any
participant can request start. Notifications are asynchronous and ordered per
leg. Server-buffered bytes and Target disk writes continue while user-paused.
Start cannot bypass Server's pressure gate and uses
the same data socket without a new handshake or seek. A registered sender
change or receiver request returns ``True``, not a peer ACK. Known duplicates
return ``False`` for ordinary legs; relay receiver requests may repeat because
another participant's opposite request can still be in flight. A preset
sender pause returns ``False`` before registration. Pause is runtime-only. See
:ref:`transfer-pause-and-reconnect` for details. The upload value's ``peer``
is the original receiver; ``resume_offset`` continues the interrupted file from
the byte the receiver already holds.

For a forwarded upload, the server forwards the existing final data-ACK marker
after the full input stream and all accepted target ACKs complete. The source
Expand All @@ -855,9 +930,9 @@ marker. The wait is 30 seconds plus 10 seconds per 100 MiB; a long pause or
slow delivery can outlast it, and a lost ACK conservatively retains state. No
separate receipt or retry protocol is used. Direct transfer and public-key
confirmation remain data-ACK based. The server retains the upstream/child IDs
and full target mapping. Transient source locks serialize different target
requests on one upload and coalesce concurrent duplicates; they hold no file
metadata and are released after the last in-flight request.
and full target mapping. Transient ownership admits one recovery per upstream
FID; another target's request is rejected while that source is live or reserved.
After cleanup it must be requested again; no active-source queue is retained.

We recommend that for client-to-server file transfers,
you use the console commands (``/file``, ``/file_folder``,
Expand Down Expand Up @@ -992,8 +1067,8 @@ is as follows:

5.1. The transfer pause and reconnect APIs (see
:ref:`transfer-pause-and-reconnect`):
- `pause_trans` / `start_trans`
- `pause_trans_remote` / `start_trans_remote`
- `pause_trans(fid, peer=None, *, trans_type, direction)` /
`start_trans(fid, peer=None, *, trans_type, direction)`
- `resume_trans`

6. The client console commands (interactive mode):
Expand Down
82 changes: 67 additions & 15 deletions docs/source/Network_APIs/TCP_Server_APIs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ The parameters of the ``__init__`` method are as follows:
user-supplied RSA keys. Both files must exist, parse as PEM, and pair
with each other; an invalid pair is silently ignored and the default
key lookup is used instead (``None`` keeps the default lookup).
- ``max_mem_buff``: The in-memory buffering ceiling of the forward pump,
given in MiB and stored as bytes (``self.max_mem_buff``); past it the
uploading peer is told to pause. The default is 2048.
- ``max_mem_buff``: The forward pump's queued-byte high-water mark, in MiB and
stored as bytes. The reader waits for low-water drain, applying TCP flow
control independently of user pause/start. One fan-out enqueue may overshoot;
it is not a total process RAM ceiling. The default is 2048.
- ``recv_messages_chunk_len``: The number of bytes read per receive on the
control connection, covering commands and messages. The default is 4096.
- ``file_transfer_chunk_len``: The chunk size in bytes for streaming a file
Expand Down Expand Up @@ -672,10 +673,21 @@ Built-in client commands include:
- ``/crypto_mode <0|1>``, ``/crypto_pub_key``, ``/crypto_pub_key_request``:
internal handshake messages of the encrypted channel.
- ``/forward_item``, ``/forward_send_msg``, ``/pause_trans``, ``/start_trans``:
internal forwarding/transfer-control messages. ``/pause_trans <fid>`` and
``/start_trans <fid>`` hold and release the outbound transfer to the requesting
peer. Local calls require ``pause_trans(fid, peer)`` / ``start_trans(fid, peer)``;
the receiver scope is required even when arming a future transfer.
internal forwarding/transfer-control messages. Control lines are strictly
``/pause_trans <fid> <type> <issuer_role>`` and
``/start_trans <fid> <type> <issuer_role>``. Local calls require
``pause_trans(fid, peer, *, trans_type, direction)`` /
``start_trans(fid, peer, *, trans_type, direction)``. ``trans_type`` is
``0`` for ordinary transfers or ``1`` for relay legs; ``direction`` is the
local data role (``0`` sender, ``1`` receiver). The wire role describes the
issuer. Ordinary senders decide their leg state; relay Source decides the
whole route. Server calls are requests, and Source notifications are projected
onto child fids. Buffered bytes keep draining to Targets during user pause.
Start uses the same sockets and cannot bypass the local pressure gate.
``True`` means registration, not peer ACK; relay requests may repeat.
Known duplicates on ordinary legs return ``False``. A preset
sender pause returns ``False`` before registration. ``peer`` is required.
See :ref:`transfer-pause-and-reconnect` for control identity and cleanup.
Forwarded requests carry a source-allocated ``fid=<upstream fid>``. A new
upload omits ``resume=``; replay includes ``resume=<offset>``, including zero,
and must match an existing route before the Server reuses its upstream and
Expand All @@ -688,12 +700,52 @@ Built-in client commands include:
sender, translates to the stable upstream fid, and forwards the same command.
Unknown or mismatched identities never fall back to another route
(see :ref:`transfer-pause-and-reconnect`). No fid kind selects this branch.
- Forwarded transfers return the existing final data-ACK marker upstream only
after the current input stream is complete and every accepted target has
acknowledged. The normal send path reframes the marker for the upstream hop;
the route is dropped only after the ACK write succeeds. A failed write keeps
the completed route for recovery. This does not change ordinary sender or
server receive-ACK behavior.
Ordinary local uploads recover only when no live, paused, negotiating or
finalizing owner holds that scoped object. ``resume_trans`` returns ``True``
for a newly reserved worker; busy and duplicate calls return ``False`` without
closing or queueing. Other objects remain independent. Use ``start_trans``
to continue an existing paused connection.
A valid forwarded receive request instead records the actual Target control
socket and nonnegative numeric disk offset, read after its old receive owner
and file have closed. No ``?`` query or ``None`` offset is supported.
Server waits for all old owners of that original route, including listeners,
negotiations, reader/writers and ``finally``, to finish naturally. It does
not force teardown, start a paused route, wait on unrelated fids or block the
control reader/event loop while holding a global lock.
It then opens an independent Source-to-Server-to-Target stream per Target,
preserving upstream/child fids, full paths and persistence schema. Offsets,
runtime progress, resources and failures are independent. Bodies run in
parallel; a late Target does not wait for another body or final ACK. Only
port negotiation on the same actual control socket plus fid is briefly
serialized, never the body or ACK wait. One Target has at most one active
recovery owner; completed Targets are not resent.
Source remains the global pause/start arbiter; shared control state is
released only by its last owner. Backpressure remains independent of user
start. Offline Source requests stay pending. A Target disconnect drops its
old-socket pending request; its new session must submit a fresh actual disk
offset after old receive cleanup. One stream's failure does not cancel
siblings or automatically retry. A real Target ACK updates only its own
completion; intermediate success cannot fabricate an overall route ACK.
Strict same-session/fid generation isolation remains deferred; C5 self-target
rejection and C6 combined acceptance remain later stages.
- After all accepted Targets really ACK, a ``False``/exception writing the
upstream final ACK retains the existing route for C4 reconfirmation. Source
explicitly calls ``resume_trans(original_target, upstream_fid, total)``;
``True`` reserves a worker, not completion. Live owners, including ``finally``,
reject this explicit operation as busy; retry explicitly after cleanup.
Server validates the original Source and accepted Target using the existing
``originator``/``targets``/``completed`` route. Only a Source-to-Server data
connection is opened: the normal port reply and full header are required,
with offset exactly equal to this header's size, zero body bytes and the
existing final ACK. No new Target leg is opened; Targets may be offline.
Another false/exception retains route and Source shared upload row. Success
removes only the owning route after the ACK write; Source removes its row
only on observing the ACK. Existing snapshots still require explicit recovery,
without automatic retry, TTL, new Server byte-progress/terminal table or
schema/wire fields. Historical name/size/hash are not validated; a successful
ACK write lost before Source receives it remains ambiguous. Ordinary transfers, C3
and global pause/start are unchanged, as are Source ``file_transfer_mode``
and ``_send_file``. See :ref:`transfer-pause-and-reconnect` for details.
- ``/server_file_transfer_port <port> <fid> <stamp>``: the four-token listener
reply for ordinary, public-key and relay legs. Replies match the actual control
socket and FID (the source-provided upstream FID for relay upload). Each
Expand Down Expand Up @@ -1175,8 +1227,8 @@ are as follows:

5.1. The transfer pause and reconnect APIs (see
:ref:`transfer-pause-and-reconnect`):
- `pause_trans` / `start_trans`
- `pause_trans_remote` / `start_trans_remote`
- `pause_trans(fid, peer, *, trans_type, direction)` /
`start_trans(fid, peer, *, trans_type, direction)`
- `resume_trans`

6. The server console commands:
Expand Down
Loading
Loading