pool: release local handlers before cooperative shutdown completes

[raphael]

Close and RemoveWorker can return success without calling JobHandler.Stop for accepted local jobs. Their Stop calls currently run inside saved-job requeue processing. When another node already owns requeue, lease acquisition returns without running those calls. Local cleanup after distributed pool cleanup also ends worker intake without releasing accepted handlers. Applications can therefore receive successful closure while their handler-owned work remains active.

This change makes each worker own local handler release independently of distributed recovery. The omission is established from the source paths; runtime reachability and frequency have not been measured.

Handler release and delivery outcomes

Workers stop accepting deliveries, wait for intake to finish, and call Stop for every locally accepted job. Successful Stops remove local execution records; failed Stops remain discoverable and their wrapped errors are returned. Close, RemoveWorker, and local cleanup after completed or lost distributed resources use that same release phase. Success requires every accepted handler to have released its work; Pulse cannot forcibly stop application goroutines.

Nodes join dispatch outcomes before processing saved jobs. Ordinary Close and RemoveWorker leave saved worker records available to the node holding the requeue lease. They no longer independently delete records that another recovery owner still needs.

A delivery that reaches a stopped worker before handler acceptance returns the existing ErrRequeue, without calling Start. Both ordinary acknowledgement and dispatch-ID outcome recording leave that delivery pending. Genuine handler Start errors keep their existing terminal classification after failed-start cleanup succeeds; cleanup failure still makes the delivery retryable.

Shutdown cleanup and errors

Shutdown has one worker-owned handler release phase. Node-owned cleanup then reads saved worker job keys directly from storage, deletes their payloads, and removes worker records only after payload cleanup succeeds. It derives remaining cleanup from existing saved records, so a storage retry does not repeat a successful Stop.

Event	Result after this change
Local Stop fails	Return the Stop error; retain failed jobs and the worker for another attempt.
Saved-job read, decoding, or payload deletion fails	Return the cleanup error; retain worker records and leave normal local closure unfinished.
Saved record is missing or contains []	Preserve successful empty-job cleanup.
Present record is JSON null or contains a non-string member	Reject it instead of treating it as an empty or altered job list.
Cleanup retries after successful Stops	Read saved records again without repeating those Stops.

Valid string arrays retain their contents, order, and duplicates. The reader adds no key normalization or new stored format. Existing distributed detach errors remain errors. Close stops new node work after its cleanup check succeeds and local closure begins. IsStopped describes stopped intake, while IsClosed describes local node closure; it does not certify pool-wide Shutdown completion. Existing cached teardown-error behavior after distributed resources disappear is preserved.

Compatibility and verification

There is no new public API, caller protocol, dependency, Redis script, or wire/storage format. No data migration or additional deployment order is required; existing upgrade rules remain unchanged. Older binaries retain their old local shutdown behavior. Rolling back needs no data conversion and restores that behavior.

The following command passed on Go 1.26.3:

GOTOOLCHAIN=local GOPROXY=off GOSUMDB=off GOWORK=off GOTELEMETRY=off go test -race ./pool -run '^TestLocalShutdown' -count=1 -timeout=45s -v

All 16 focused tests passed, including 21 saved-record cases. They exercise production local release, admission/completion, decoder, and closure algorithms with synthetic jobs and in-process saved-data operations. Coverage includes joining intake, concurrent Stops, failed Stop retry, successful Stop followed by cleanup retry, both delivery outcome routes, preserved genuine Start errors, and invalid records preventing cleanup completion. Formatting and whitespace checks passed.

No Redis client, network recovery, or fault experiment was exercised. The tests do not prove lease acquisition, Lua execution, requeue publication, distributed detach, crash recovery, or the pool-wide Shutdown barrier. The configured CI workflow runs static checks and Redis-backed race tests on Go 1.26 and 1.27; those checks have not been run for this candidate.

Review Worker.stopLocal, startJob/startJobLocked, and completeEvent in pool/worker.go; Node.closeWithCleanup, cleanupForClose, closeAfterDistributedLoss, and decodeWorkerJobKeys in pool/node.go; and the contracts and synthetic proof in pool/README.md and pool/local_shutdown_test.go.