Align observability and message result contracts

[cardmagic]

Summary

Add portable observability for cardmagic/solid-objects-js#42 and align message behavior with JavaScript #59. Both runtimes expose versioned JSON events, metric samples, authorized local observers, and bounded actor diagnostics without exposing private payloads.

• Guard personalized payloads against state mutation, staged work, and application database writes. Evaluate each payload on a fresh actor from the same committed snapshot, so a failed projection cannot contaminate another subscriber or payload. Generated defaults are captured once for that snapshot. Honor the configured max_payload_bytes, including UTF-8 boundaries and limits above 1 MB.
• Preserve timeout waitingOn, activationOwnerId, and string activationGeneration fields. Normalize portable wait reasons to the shared camelCase values while preserving native Ruby diagnostics.
• Use a yielding SQLite busy handler on each background transaction attempt, including retries and reconnections. This lets the lock holder commit on Rails 7.1/7.2 and preserves synchronous deadlines.
• Reject state mutation and durable work staged by queries or observable projections, including individual snapshot reads. Violations raise terminal QueryMutatedState, matching JS; normal operation intents survive pure projections. Guards compare complete staged work, so replacing an intent without changing the count also fails.
• Add instrumentation, actor observer hooks with unsubscribe support, and administration-authorized diagnostics for mailboxes, retries, reminders, outbox, and recovery failures. Preserve Active Support notifications and isolate subscriber failures.
• Reauthorize every message status, result, and outcome read against the original operation or query and arguments. Validate reference identity and apply the caller's current authorization context.
• Retain bounded JSON results for background operations. Result reads raise persisted rejection or permanent-failure errors; outcome reads preserve structured terminal outcomes. Share result decoding with synchronous invocation.
• Normalize portable polling intervals to milliseconds and symbolic reasons to strings. Cover transmit staging order after retry and reject explicit null arguments through the shared compatibility fixture.
• Share reserved JSON property fixtures with JS and document the existing 191-character Ruby / 255 UTF-16 code unit JS keyed reminder limits. Replace obsolete manual-SQL transmit recovery instructions with the authorized retry/redrive API.
• Share the portable event contract compatibility/telemetry-events.json with JS. It holds the attribute allowlist and the exact keys of each core SQL event, and both test suites check their events against it. Message events now send operation and deliveryMode, message.failed sends retryable and outcome, and commit action events send the message fields and commitAction. reminder.enqueued, outbox.age, and sync.enqueue_timeout send the same keys as JS, and polling intervals are integers.
• Rename solid_objects.payload_broadcast_failed to solid_objects.payload_broadcast.failed, the dotted form that every other event uses.
• Log solid_objects.instrumentation.failed when an exporter or observer raises; Ruby discarded these errors before. Observers need a block, and a process accepts at most 1,000 observers, as in JS.
• Add a Ruby setup sample to the shared observability guide, and mark the Durable Objects observer limit as JavaScript only.
• Update generated RBS, architecture and authorization docs, the roadmap, and changelog. The paired JS PR updates the parity ledger and adds persisted effect ordering.

Examples

These snippets extend an application's registered ShoppingCart actor. operator is the current authenticated caller; the application's message and administration policies must authorize it.

Send portable events and metric samples to your logger

SolidObjects.configure do |configuration|
  configuration.instrumentation = ->(event) do
    Rails.logger.info(JSON.generate(event))
  end
end

Events include schemaVersion, actor/message correlation, safe attributes, and metric samples. Ruby and JS emit the same JSON fields, including polling intervals in milliseconds. Arguments, state, results, and exception text are excluded; exporter failures do not fail actor work. Existing Active Support notifications remain available.

Watch one actor and inspect its queues

cart = ShoppingCart.ref("demo-cart")
stop = cart.on("message.retry", authorization_context: operator) do |event|
  Rails.logger.info(JSON.generate(event))
end

summary = cart.diagnostics(authorization_context: operator, limit: 50)
pp summary.fetch("mailbox"), summary.fetch("outbox"), summary.fetch("recoveryFailures")
stop.call

Administration policies must allow :observe and :inspect on actor_diagnostics. Each category exposes sampled, truncated, and oldestAgeMilliseconds. Use cart.observe(authorization_context:) { |event| ... } for all of this actor's events. Observers are local to the current process.

Recover a background result after losing the original handle

Assume checkout(order_id:) returns { "order_id" => order_id }. Enqueue it with a stable key:

cart.async(
  idempotency_key: "checkout-42",
  authorization_context: operator
).checkout(order_id: 42)

After a worker processes it, a later request can recover the result:

message = cart.find_by(
  idempotency_key: "checkout-42",
  authorization_context: operator
)
if message
  result = message.result(authorization_context: operator)
  outcome = message.outcome(authorization_context: operator)
  pp result, outcome.status, outcome.attempts
end

A completed result is { "order_id" => 42 }. Background results are now retained, validated as JSON, and bounded by max_result_bytes. Every read checks current authorization against the original operation and arguments. result raises SolidObjects::Rejected or SolidObjects::MessageFailed for terminal failures; outcome exposes them as data.

Keep two transmits in staging order through retries

Inside an actor operation, stage two calls to its server twin's declared append operation:

transmit.append(value: 1)
transmit.append(value: 2)
nil

With the transmit handler registered, the receiver applies 1 before 2, even if delivery of 1 initially fails. Returning nil explicitly avoids retaining an incidental result. This PR pins Ruby's existing order with a retry regression test; the paired JS PR fixes its ordering to match. Both ingest paths reject explicit null arguments and deduplicate repeated envelopes.

Diagnose why a synchronous call timed out

Filter solid_objects.sync.timeout events to see whether an activation, an earlier
message, a paused actor, or database contention prevented progress. The Ruby and
JS SQL runtimes now retain the same fields and reason values. The Durable Objects
host sends no sync.timeout event; its call timeouts report waitingOn: "unknown".
For example, an event can include:

{
  "name": "solid_objects.sync.timeout",
  "attributes": {
    "waitingOn": "activationHeld",
    "activationOwnerId": "worker-1",
    "activationGeneration": "7"
  }
}

Unknown activation fields are null. These are correlation fields, not metric labels.

Compatibility

Queries, observables, and personalized payloads must be pure: they cannot change actor state or stage effects, recovery checks, commit actions, reminders, or outbound messages. Query and observable violations fail without retries; a payload violation is confined to that payload and instrumented. Projection guards compare state and staged work before and after evaluation. Payloads also prevent application database writes and allocate an isolated actor for each evaluation.

Message-reference reads now accept authorization_context: and reevaluate policy on every read. Callers using protected actors must supply their current context.

All operation return values, including background calls, must serialize as JSON and fit max_result_bytes. Return nil when an operation needs no result. Previously discarded background results cannot be recovered retroactively. No Ruby database migration is required.

Observers are local to the current process. Diagnostic samples are bounded observations; they do not provide a transactional snapshot across the fleet.

Active Support subscribers to solid_objects.payload_broadcast_failed must subscribe to solid_objects.payload_broadcast.failed. The Active Support payload keeps payload_name; the portable event names it payload. observe and on raise ArgumentError without a block or after 1,000 observers in one process.

solid_objects.activation.started now fires before the activate hook. Before, it fired after a successful hook. The new activation.completed event takes that meaning, and activation.failed reports a failed hook. JavaScript changes the same events in the paired PR. Move a subscriber that reads activation.started as a finished activation to activation.completed.

Active Support payloads no longer carry error_message for commit_action.failed, activation.deactivation_failed, supervisor.monitor_failed, supervisor.retention_failed, supervisor.redrive_failed, and wake_up.failed. The solid_objects.worker.error log entry also omits it. Each keeps error_class. Exception text can contain actor state; JavaScript already reports only the error name.

Validation

• Sync deadline test budget (08cd427): the Rails 7.1 job in run 36877758705 failed with Timeout::Error at synchronous_invocation_test.rb:422, because one 0.25 s deadline covered the actor start and the lock wait. A 0.3 s stall before the actor claim reproduces that error with the old budget. With a 1 s budget, 0.3 s and 0.6 s stalls pass. When SyncDeadline.expired? never returns true, the changed test fails at line 443 on Rails 7.1.6 and 8.1.3.1. The whole file passed three random orders on Rails 7.1.6; bundle exec rake ran 864 tests with 0 failures/errors and 28 skips.
• Shared event contract: before the fix, 13 tests failed and 2 had errors. Examples: activation.started attributes expected ownerId; ArgumentError expected but nothing was raised for a blockless observer and for observer 1,001; the polling JSON showed 100.0 where the contract needs 100; Expected [] to include {event: "solid_objects.instrumentation.failed", ...} for a failed exporter. All pass after the fix.
• Latest head (da1f00e, docs and changelog only on top of 30bf741), local SQLite: bundle exec rake ran 863 tests and 3,265 assertions with 0 failures/errors and 28 existing skips. Standard, RuboCop, RBS validation, Steep, and Brakeman passed.
• New regressions pass against PostgreSQL and MySQL with both mysql2 and Trilogy: 92 tests / 446 assertions per adapter, no failures/errors/skips.
• Rails 7.1.6 and 7.2.3 with sqlite3 2.9.6: full suite passed (853 tests / 3,044 assertions, 39 existing skips). This is the previously failing compatibility combination.
• Reversing the five Ruby implementation files reproduces 18 failures and two errors. Observed failures include SolidObjects::QueryMutatedState expected but nothing was raised, timeout metadata becoming {}, a fixed 1 MB cap rejecting an explicitly larger limit, and SQLite3::BusyException: database is locked. Restoring the implementation passes.
• Generated-default reversal: removing snapshot default capture makes two payloads receive different random IDs; restoring it preserves the original snapshot value.
• The SQLite regression coordinates the lock holder and contender with queues and the actual busy callback. It reproduces the old failure in about 0.15 seconds; no larger hang budget is needed.
• Ruby 3.3.9 / Rails 8.1 validation: 853 tests, 3,044 assertions, 0 failures/errors, 39 existing adapter/environment skips. Standard, RuboCop, RBS validation, Steep, and Brakeman passed.
• PostgreSQL purity and state-commit checks: 25 tests, 227 assertions, no failures/errors/skips.
• Replacement reversal: restoring the count-only guard produces MessageFailed expected but nothing was raised for both effect and commit-action replacement. Restoring the fix passes all 16 purity tests.
• Purity reversal: restoring the old actor/executor code produces 13 expected failures; restoring the fix passes all 37 focused purity/serialization tests.
• PostgreSQL result lookup, telemetry, and transmit checks: 78 tests, 232 assertions, no failures/errors/skips.
• Reversal checks: restoring the old message and polling implementations produces seven expected regression failures. Restoring these fixes passes the focused suite.
• Earlier observability review regressions cover subscriber isolation, optional outbox measurement, outcome-only duration samples, and combined diagnostic sample limits.

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds portable telemetry and reauthorizes message-reference reads.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR adds portable telemetry and authorized diagnostics, reauthorizes message-result reads, retains background results, and enforces read-only queries and projections. The latest commit replaces count-only staged-work checks with deep snapshots and adds replacement regressions.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Actor operation] --> B[Stage durable work]
  B --> C[Snapshot staged intents]
  C --> D[Evaluate observables]
  D --> E{State and intents unchanged?}
  E -->|Yes| F[Commit operation and staged work]
  E -->|No| G[QueryMutatedState; do not commit]

Loading

_{Reviews (5) · Last reviewed commit: "fix: Detect replaced work in projections"}

[cardmagic]

@greptileai Please review the current head, 7149e9c. The existing summary still references the initial commit.

• Optional outbox measurement now runs inside Telemetry.outbox with failure isolation around every lookup and emission. A regression verifies that a measurement exception still lets the handler run and complete its effect; restoring the unsafe implementation makes that test fail.
• Broadcast delivery now uses the same helper and emits its age sample.
• The diagnostic limit deliberately caps the combined category across sources. With limit 1, an effect plus a broadcast reports sampled 1 and truncated true. Ruby and JavaScript share this contract; the docs state it explicitly and a regression covers it. The metric omits truncated samples rather than reporting them as exact depth.
• Message duration samples now appear only on outcomes, matching JavaScript.

All current CI jobs pass. Please reassess the full diff and update the summary for this head.

[cardmagic]

@greptileai Please review current head 01b2981 and update the summary for this commit. The parity fixes reauthorize every message read, retain bounded JSON results for background operations, share terminal result errors with synchronous calls, normalize polling telemetry, and update compatibility tests and documentation. The full Ruby suite and focused PostgreSQL checks pass; reversal checks reproduce seven failures with the old implementations. Please review the full diff, including these compatibility changes.

[cardmagic]

@greptileai Please review current head 4530819. The latest change rejects state mutation and all staged durable work in queries and observable projections, including individual snapshot projections, with terminal QueryMutatedState. Pure projections preserve work staged by normal operations. Reversal reproduces 13 failures; all 830 Ruby tests, lint, RBS, Steep, Brakeman, and focused PostgreSQL checks pass. Shared JSON fixtures and corrected reminder/retry documentation complete the audit follow-up. Please review the full diff and update the current-head summary.

[cardmagic]

@greptileai Please review the latest commit d9667bd. The observable replacement finding is fixed by comparing deep snapshots of all staged work, with effect and commit-action reversal regressions. The equivalent JS guard is fixed too. All 832 Ruby tests and static/security checks pass.