From 0dcbcbd678414820768bbb72d01fe062256bf69f Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 08:27:29 -0700 Subject: [PATCH 01/11] feat: Add portable actor observability Match JavaScript telemetry and actor diagnostics using native Active Support notifications and Active Record scopes. Isolate subscriber failures while preserving application exceptions, and keep private error text out of default logs. Refs cardmagic/solid-objects-js#42 --- CHANGELOG.md | 5 + README.md | 1 + docs/observability.md | 123 ++++++++++++++++++ docs/roadmap.md | 3 + lib/solid_objects.rb | 2 + lib/solid_objects/activation.rb | 9 +- lib/solid_objects/actor_channel.rb | 8 ++ lib/solid_objects/configuration.rb | 3 + lib/solid_objects/diagnostics.rb | 86 ++++++++++++ lib/solid_objects/effect_executor.rb | 4 + lib/solid_objects/executor.rb | 20 ++- lib/solid_objects/instrumentation.rb | 23 +++- lib/solid_objects/log_subscriber.rb | 2 +- lib/solid_objects/mailbox.rb | 2 + lib/solid_objects/reference.rb | 17 +++ lib/solid_objects/reminder_scheduler.rb | 4 + lib/solid_objects/state_snapshot.rb | 1 + lib/solid_objects/supervisor.rb | 9 +- lib/solid_objects/telemetry.rb | 73 +++++++++++ .../wake_up_adapters/postgresql.rb | 3 +- lib/solid_objects/wake_up_adapters/redis.rb | 3 +- lib/solid_objects/worker.rb | 3 +- .../lib/solid_objects/actor_channel.rbs | 3 + .../lib/solid_objects/configuration.rbs | 10 +- .../lib/solid_objects/diagnostics.rbs | 25 ++++ sig/generated/lib/solid_objects/executor.rbs | 5 + sig/generated/lib/solid_objects/reference.rbs | 9 ++ sig/generated/lib/solid_objects/telemetry.rbs | 18 +++ test/integration/actor_channel_test.rb | 5 + .../actor_code_write_guard_test.rb | 3 + test/integration/process_lifecycle_test.rb | 3 + test/integration/telemetry_test.rb | 121 +++++++++++++++++ 32 files changed, 582 insertions(+), 24 deletions(-) create mode 100644 docs/observability.md create mode 100644 lib/solid_objects/diagnostics.rb create mode 100644 lib/solid_objects/telemetry.rb create mode 100644 sig/generated/lib/solid_objects/diagnostics.rbs create mode 100644 sig/generated/lib/solid_objects/telemetry.rbs create mode 100644 test/integration/telemetry_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c049c8..1a6e90f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## Unreleased + +- Add portable telemetry, isolated observer hooks, metric definitions, and bounded authorized actor diagnostics matching JavaScript. + + ## 0.16.0 - 2026-09-23 - Find a message whose reference a caller lost. diff --git a/README.md b/README.md index 7dcda16..118cc09 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ Exactly once is not hiding in a more advanced configuration. Read the - [Five-minute Rails guide](https://solidobjects.dev/5min/rails) - [Choosing Solid Objects](docs/fit.md) - [Operations and recovery](docs/operations.md) +- [Observability and diagnostics](docs/observability.md) - [Reminders](docs/reminders.md) - [Reactive ERB](docs/realtime.md) - [Detailed architecture](docs/architecture.md) diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 0000000..061c1aa --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,123 @@ +# Portable observability + +Configure `instrumentation(event)` to receive structured events. No exporter SDK +is required. The same JSON envelope is emitted by Ruby, SQLite, PostgreSQL, MySQL, +and the Durable Objects host. Existing JavaScript `name`, `occurredAt`, and +`attributes` fields remain available. Ruby's Active Support notifications remain +available with their existing snake_case payloads. + +## Schema version 1 + +Every event has `schemaVersion`, `name` (prefixed with `solid_objects.`), +`occurredAt` (UTC ISO 8601), `adapter`, `actorType`, `actorId`, `incarnation`, +`revision`, `messageId`, `attempt`, `attributes`, and `metrics`. +Unavailable identifiers are null; `attempt` is zero outside a message attempt. +An incarnation identifies a persisted actor instance, independently of its lease +generation. Revisions and IDs are strings. Process-wide events have null actor +identity. A message ID or revision correlates actor work where applicable. + +Only known scalar metadata fields enter the portable envelope. Arguments, state, +results, credentials, backtraces, exception text, nested provider data, and unknown +attributes are excluded. Actor IDs remain correlation data: applications should +use opaque actor identifiers and apply their own retention policy to event logs. +Events and metric samples are immutable. Throwing observers, rejected observer +promises, and a failing instrumentation error logger cannot change a turn's +result. Delivery is best effort and synchronous callbacks should be short; +JavaScript does not await exporters. Telemetry is not a durable audit trail. + +| Event | Meaning | +| --- | --- | +| `activation.started/completed/failed` | Local actor activation hook lifecycle | +| `message.started/completed/failed/rejected` | One attempt's execution outcome | +| `message.retry` | Failed attempt durably queued for another attempt | +| `dead_letter.created` | Message exhausted retries or failed permanently | +| `mailbox.depth` | On-demand diagnostic sample; exact depth only if not truncated | +| `reminder.enqueued` | Due reminder dispatch; lateness is measured from due time | +| `outbox.age` | Delivery observation; age is time since the item's current availability time | +| `recovery.reclaimed` | A previously claimed, interrupted message begins another attempt | +| `recovery.completed/failed` | Durable effect recovery callback commits or enters the dead-letter queue | +| `snapshot.read` | Authorized snapshot constructed without exposing its contents | +| `realtime.connected/disconnected` | Actor subscription added or removed | + +Events describe local observations. Concurrent deletion, crashes, and failed +exporters can omit events. Never infer exactly-once delivery from event counts. +Additional existing runtime events retain their names. + +## Metrics and tracing + +Metrics are sample descriptions. Exporting them is opt-in: the runtime does not +register meters, allocate per-actor metric series, or install a vendor SDK. + +| Name | Kind | Unit | Aggregation | +| --- | --- | --- | --- | +| `solid_objects.events` | counter | `1` | Sum one per event | +| `solid_objects.duration` | histogram | `ms` | Distribution of observed attempt duration | +| `solid_objects.reminder.lateness` | histogram | `ms` | Distribution of reminder dispatch delay | +| `solid_objects.outbox.age` | histogram | `ms` | Distribution of delivery delay since availability | +| `solid_objects.mailbox.depth` | gauge | `1` | Last exact sampled actor depth; omit truncated samples | + +Labels contain only event name, adapter family, and declared actor type. Keep the +actor type registry finite. Never add actor ID, incarnation, message ID, operation +arguments, request IDs, or error text to metric labels. A gauge without an actor +label represents the most recently observed actor; it is not total fleet backlog. +For tracing, correlate start/outcome events using `adapter`, `incarnation`, +`messageId`, and `attempt`, and close or expire spans when no outcome arrives. + +## Actor observers and diagnostics + +```ruby +cart = ShoppingCart.ref("demo-cart") +stop = cart.observe(authorization_context: operator) do |event| + logger.info(event.to_json) +end +summary = cart.diagnostics(authorization_context: operator, limit: 50) +stop.call +``` + +Ruby uses `reference.observe(authorization_context:) { |event| ... }` and +`reference.diagnostics(authorization_context:, limit: 50)`. Stop observing by +calling the returned proc. `on` filters one event name, such as `message.retry`. +Observers receive only this actor's events in the current runtime/process; they +are not subscriptions to workers on other hosts. Dispose them when the caller's +session ends or authorization is revoked. JS limits local observers to 1,000. +For remote Durable Objects, configure `instrumentation` on the actor host and +filter by actor identity there; process-local reference observers raise +`UnsupportedCapability`. Remote `reference.diagnostics` is supported. + +Both APIs default to denied. Set `authorizeAdministration` / `authorize_administration` +to allow action `observe` or `inspect`, resource `actor_diagnostics`, and resource ID +`JSON.stringify([actorType, actorId])`. Ruby receives a symbol action. Authorization +runs before reading summaries or registering observers; possessing an actor ID +confers no permission. + +Diagnostics read at most `limit + 1` rows per queue source, with a hard limit of +100. Each category returns `sampled`, `truncated`, and `oldestAgeMilliseconds`. +The last value measures nonnegative time since availability (or terminal failure +for recovery callbacks); future reminders have zero age. No payloads or row +identifiers are returned. Samples are observations across several queries, +not an atomic fleet snapshot. Large queues can still require database scanning; +the bound limits materialized rows and response size, not query execution time. + +Categories are mailbox (ready and claimed), outbox (pending and processing effects +and broadcasts), reminders (scheduled and paused), retries (failed messages still +eligible to run), and recoveryFailures (dead internal effect recovery +callback messages). Durable Objects does not implement process-heartbeat effect recovery; +its recoveryFailures category is empty. Recovery failures are durable records, +not a history of transient database or exporter exceptions. + +## A query that works across adapters + +Write one envelope per line to `events.jsonl`, using the same instrumentation hook +with SQLite and PostgreSQL. This query reports completed attempts by adapter: + +```sh +jq -s 'map(select(.name == "solid_objects.message.completed")) + | group_by(.adapter) + | map({adapter: .[0].adapter, completed: length, + mean_ms: (map(.attributes.durationMilliseconds) | add / length)})' events.jsonl +``` + +A dashboard can chart the event counter by adapter and outcome and the duration, +reminder lateness, and outbox age distributions using exactly the same fields. +Compare workloads with the same actor types and sampling policy. Counts measure +observations and cannot replace a database query for authoritative queue state. diff --git a/docs/roadmap.md b/docs/roadmap.md index db7e4f2..06b10ac 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -2,6 +2,9 @@ ## Implemented and tested +- Portable telemetry with a shared JSON schema, metric samples, isolated observer + callbacks, and bounded authorized actor diagnostics. See [observability](observability.md). + - Rails engine, install generator, migration, and CLI - Explicit actor registry, references, JSON state, and state migrations - Fluent direct synchronous RPC, configured `sync`, and durable `async` diff --git a/lib/solid_objects.rb b/lib/solid_objects.rb index d74df6d..9c84640 100644 --- a/lib/solid_objects.rb +++ b/lib/solid_objects.rb @@ -16,6 +16,8 @@ require "solid_objects/callable_keywords" require "solid_objects/configuration" require "solid_objects/instrumentation" +require "solid_objects/telemetry" +require "solid_objects/diagnostics" require "solid_objects/log_subscriber" require "solid_objects/serialization" require "solid_objects/context" diff --git a/lib/solid_objects/activation.rb b/lib/solid_objects/activation.rb index d74547a..c7ba0d9 100644 --- a/lib/solid_objects/activation.rb +++ b/lib/solid_objects/activation.rb @@ -24,15 +24,19 @@ def initialize(lease:) @actor = build_actor(instance) @last_used_at = monotonic_now @pass_exhausted = false + SolidObjects.instrument(:"activation.started", instance_id: instance.id, actor_type: instance.actor_type, actor_id: instance.actor_id, generation: lease.generation) actor.activate SolidObjects.instrument( - :"activation.started", + :"activation.completed", instance_id: instance.id, actor_type: instance.actor_type, actor_id: instance.actor_id, owner_id: lease.owner_id, generation: lease.generation ) + rescue => error + SolidObjects.instrument(:"activation.failed", instance_id: lease.instance_id, actor_type: instance&.actor_type, actor_id: instance&.actor_id, error_class: error.class.name) + raise end # @rbs () -> Integer @@ -104,8 +108,7 @@ def deactivate actor_id: actor.actor_id, owner_id: lease.owner_id, generation: lease.generation, - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) SolidObjects.configuration.logger.error( "SolidObjects activation deactivation failed " \ diff --git a/lib/solid_objects/actor_channel.rb b/lib/solid_objects/actor_channel.rb index 7bbf4d8..71798a0 100644 --- a/lib/solid_objects/actor_channel.rb +++ b/lib/solid_objects/actor_channel.rb @@ -49,6 +49,7 @@ def subscribed end refresh_outdated_components(snapshot) transmit_state_payloads(snapshot) + SolidObjects.instrument(:"realtime.connected", actor_type: reference.actor_type, actor_id: reference.actor_id, instance_id: snapshot.instance_id, revision: snapshot.revision) rescue KeyError, JSON::ParserError, InvalidStreamToken, @@ -57,6 +58,13 @@ def subscribed reject_and_report(reject_reason(error), actor_type:, actor_id:, error:) end + # @rbs () -> void + def unsubscribed + return unless reference + + SolidObjects.instrument(:"realtime.disconnected", actor_type: reference.actor_type, actor_id: reference.actor_id) + end + private attr_reader :reference, diff --git a/lib/solid_objects/configuration.rb b/lib/solid_objects/configuration.rb index 5187c9d..6f320ae 100644 --- a/lib/solid_objects/configuration.rb +++ b/lib/solid_objects/configuration.rb @@ -40,6 +40,7 @@ class Configuration # @rbs @broadcast_worker_count: Integer # @rbs @reminder_scheduler_count: Integer # @rbs @connects_to: Hash[Symbol, untyped]? + # @rbs @instrumentation: Proc? # @rbs @logger: untyped # @rbs @stream_signing_secret: String? # @rbs @broadcast_adapter: Proc? @@ -95,6 +96,7 @@ class Configuration :reminder_scheduler_count, :connects_to, :logger, + :instrumentation, :stream_signing_secret, :broadcast_adapter, :wake_up_adapter, @@ -159,6 +161,7 @@ def initialize @component_path_resolver = nil @component_authorization_context = ->(controller:) { controller } @payload_authorization_context = ->(connection:) { connection } + @instrumentation = nil @logger = if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger Rails.logger else diff --git a/lib/solid_objects/diagnostics.rb b/lib/solid_objects/diagnostics.rb new file mode 100644 index 0000000..8a837ed --- /dev/null +++ b/lib/solid_objects/diagnostics.rb @@ -0,0 +1,86 @@ +# rbs_inline: enabled + +module SolidObjects + class Diagnostics + # @rbs @reference: Reference + # @rbs (Reference) -> void + def initialize(reference) + @reference = reference + end + + # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + def summary(limit: 100, authorization_context: nil) + authorize!(:inspect, authorization_context:) + raise ArgumentError, "diagnostic limit must be an integer between 1 and 100" unless limit.is_a?(Integer) && limit.between?(1, 100) + + instance = Instance.find_by(actor_type: reference.actor_type, actor_id: reference.actor_id) + now = SolidObjects.database_adapter.database_now + instance_id = instance&.id + mailbox = ReadyMessage.where(instance_id:).order(:available_at).limit(limit + 1).pluck(:available_at) + + ClaimedMessage.where(instance_id:).order(:claimed_at).limit(limit + 1).pluck(:claimed_at) + outbox = Effect.where(instance_id:, status: %w[pending processing]).order(:available_at).limit(limit + 1).pluck(:available_at) + + Broadcast.where(instance_id:, status: %w[pending processing]).order(:available_at).limit(limit + 1).pluck(:available_at) + reminders = Reminder.where(instance_id:, status: %w[scheduled paused]).order(:next_run_at).limit(limit + 1).pluck(:next_run_at) + retries = ReadyMessage.joins(:message).where(instance_id:).where.not(Message.table_name => { error: nil }).order(:available_at).limit(limit + 1).pluck(:available_at) + recovery_messages = Message.where(instance_id:, delivery_mode: "internal").where("idempotency_key LIKE ?", "effect:%:recovery").select(:id) + recovery_failures = DeadLetter.where(instance_id:, message_id: recovery_messages).order(:last_failed_at).limit(limit + 1).pluck(:last_failed_at) + result = { + "actorType" => reference.actor_type, + "actorId" => reference.actor_id, + "incarnation" => instance_id&.to_s, + "revision" => instance&.state_revision&.to_s, + "adapter" => DatabaseAdapter.family(Record.connection).to_s, + "occurredAt" => now.utc.iso8601(3), + "limit" => limit, + "mailbox" => summarize(mailbox, now:, limit:), + "outbox" => summarize(outbox, now:, limit:), + "reminders" => summarize(reminders, now:, limit:), + "retries" => summarize(retries, now:, limit:), + "recoveryFailures" => summarize(recovery_failures, now:, limit:) + } + SolidObjects.instrument(:"mailbox.depth", actor_type: reference.actor_type, actor_id: reference.actor_id, instance_id:, count: result.fetch("mailbox").fetch("sampled"), truncated: result.fetch("mailbox").fetch("truncated"), depth: result.fetch("mailbox").fetch("truncated") ? nil : result.fetch("mailbox").fetch("sampled")) + Serialization.readonly_copy(result) + end + + # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def observe(authorization_context: nil, &block) + authorize!(:observe, authorization_context:) + subscription = ActiveSupport::Notifications.subscribe(/\Asolid_objects\./) do |notification| + payload = notification.payload + next unless payload[:actor_type] == reference.actor_type && payload[:actor_id] == reference.actor_id + + begin + block.call(Telemetry.event(notification.name.delete_prefix("solid_objects.").to_sym, payload)) + rescue + nil + end + end + -> { ActiveSupport::Notifications.unsubscribe(subscription) } + end + + private + + attr_reader :reference + + # @rbs (Symbol, ?authorization_context: untyped) -> void + def authorize!(action, authorization_context: nil) + allowed = SolidObjects.configuration.authorize_administration.call( + action:, + resource: "actor_diagnostics", + resource_id: [ reference.actor_type, reference.actor_id ].to_json, + authorization_context: + ) + raise Unauthorized, "actor diagnostics are not authorized" unless allowed + end + + # @rbs (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] + def summarize(timestamps, now:, limit:) + oldest = timestamps.compact.min + { + "sampled" => [ timestamps.length, limit ].min, + "truncated" => timestamps.length > limit, + "oldestAgeMilliseconds" => oldest ? [ ((now - oldest) * 1000).round, 0 ].max : nil + } + end + end +end diff --git a/lib/solid_objects/effect_executor.rb b/lib/solid_objects/effect_executor.rb index e9d089e..cdf83d3 100644 --- a/lib/solid_objects/effect_executor.rb +++ b/lib/solid_objects/effect_executor.rb @@ -137,6 +137,7 @@ def claim_next # @rbs (Effect) -> untyped def deliver(effect) + SolidObjects.instrument(:"outbox.age", instance_id: effect.instance_id, actor_type: effect.instance.actor_type, actor_id: effect.instance.actor_id, message_id: effect.message_id, attempt: effect.attempt_count, age_milliseconds: [ ((database_adapter.database_now - effect.available_at) * 1000).round, 0 ].max) return deliver_actor_message(effect) if effect.name == ACTOR_MESSAGE_EFFECT handler = SolidObjects.effect_registry.fetch(effect.name) @@ -208,6 +209,9 @@ def complete(effect, result) Mailbox.new.announce(result_message) if result_message SolidObjects.instrument( :"effect.completed", + instance_id: effect.instance_id, + actor_type: effect.instance.actor_type, + actor_id: effect.instance.actor_id, effect_id: effect.effect_id, effect_name: effect.name, message_id: effect.message_id, diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index 0df5724..d05609e 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -4,12 +4,14 @@ module SolidObjects class Executor # @rbs @activation: Activation # @rbs @message: Message + # @rbs @started_at: Float # @rbs @completion_transaction: untyped # @rbs (activation: Activation, message: Message) -> void def initialize(activation:, message:) @activation = activation @message = message + @started_at = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC) end # @rbs () -> bool @@ -24,6 +26,7 @@ def call request_id: message.request_id ) + SolidObjects.instrument(:"recovery.reclaimed", **instrumentation_payload) if message.attempt_count > 1 && message.error.nil? SolidObjects.instrument(:"message.started", **instrumentation_payload) result = invoke_actor(message_context) observable_changes = changed_observables(observables_before, actor.observable_values) @@ -156,8 +159,9 @@ def complete(result, observable_changes, state_after:, state_changed:) actor_id: message.actor_id ) end + SolidObjects.instrument_after_commit(:"recovery.completed", **instrumentation_payload) if recovery_message? report_large_state(state_after.byte_size) - SolidObjects.instrument_after_commit(:"message.completed", **instrumentation_payload) + SolidObjects.instrument_after_commit(:"message.completed", **instrumentation_payload, revision: message.sequence) SolidObjects.wake_up.signal rescue CommittedTransactionError raise @@ -224,8 +228,7 @@ def execute_commit_action(intent:, handler:, context:) SolidObjects.instrument( :"commit_action.failed", **payload, - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) raise end @@ -423,6 +426,8 @@ def fail_message(error) error_class: error.class.name, dead: ) + SolidObjects.instrument_after_commit(:"recovery.failed", **instrumentation_payload) if dead && recovery_message? + SolidObjects.instrument_after_commit(dead ? :"dead_letter.created" : :"message.retry", **instrumentation_payload) SolidObjects.wake_up.signal end @@ -520,15 +525,22 @@ def create_dead_letter(message:, error_details:, now:) ) end + # @rbs () -> bool + def recovery_message? + message.delivery_mode == "internal" && message.idempotency_key.to_s.start_with?("effect:") && message.idempotency_key.to_s.end_with?(":recovery") + end + # @rbs () -> Hash[Symbol, untyped] def instrumentation_payload { + instance_id: message.instance_id, message_id: message.id, actor_type: message.actor_type, actor_id: message.actor_id, sequence: message.sequence, attempt: message.attempt_count, - request_id: message.request_id + request_id: message.request_id, + duration_milliseconds: ((::Process.clock_gettime(::Process::CLOCK_MONOTONIC) - @started_at) * 1000).round(3) } end end diff --git a/lib/solid_objects/instrumentation.rb b/lib/solid_objects/instrumentation.rb index 29fc57e..9f3f149 100644 --- a/lib/solid_objects/instrumentation.rb +++ b/lib/solid_objects/instrumentation.rb @@ -4,7 +4,28 @@ module SolidObjects module Instrumentation # @rbs (Symbol, **untyped) { (Hash[Symbol, untyped]) -> untyped } -> untyped def instrument(event, **payload, &block) - ActiveSupport::Notifications.instrument("solid_objects.#{event}", payload, &block) + executed = false + application_error = nil + result = nil + begin + ActiveSupport::Notifications.instrument("solid_objects.#{event}", payload) do + executed = true + begin + result = block&.call(payload) + rescue => error + application_error = error + raise + end + end + rescue => error + raise application_error if application_error + + report_instrumentation_failure(event, error) unless event == :"instrumentation.failed" + result = block&.call(payload) unless executed + ensure + Telemetry.emit(event, payload) + end + result end # @rbs (Symbol, **untyped) -> void diff --git a/lib/solid_objects/log_subscriber.rb b/lib/solid_objects/log_subscriber.rb index 7f2b37c..b78a0f0 100644 --- a/lib/solid_objects/log_subscriber.rb +++ b/lib/solid_objects/log_subscriber.rb @@ -12,7 +12,7 @@ def install { event: event.name, duration_ms: event.duration.round(2) - }.merge(event.payload) + }.merge(event.payload.except(:error_message, :arguments, :state, :credentials, :response, :exception, :exception_object)) ) end end diff --git a/lib/solid_objects/mailbox.rb b/lib/solid_objects/mailbox.rb index 65c2167..0822816 100644 --- a/lib/solid_objects/mailbox.rb +++ b/lib/solid_objects/mailbox.rb @@ -79,6 +79,8 @@ def enqueue_in_transaction( def announce(message) SolidObjects.instrument( :"message.enqueued", + instance_id: message.instance_id, + attempt: message.attempt_count, message_id: message.id, actor_type: message.actor_type, actor_id: message.actor_id, diff --git a/lib/solid_objects/reference.rb b/lib/solid_objects/reference.rb index 5a52f57..2f6aa09 100644 --- a/lib/solid_objects/reference.rb +++ b/lib/solid_objects/reference.rb @@ -16,6 +16,23 @@ def initialize(actor_type:, actor_id:) freeze end + # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + def diagnostics(limit: 100, authorization_context: nil) + Diagnostics.new(self).summary(limit:, authorization_context:) + end + + # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def observe(authorization_context: nil, &block) + Diagnostics.new(self).observe(authorization_context:, &block) + end + + # @rbs (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def on(name, authorization_context: nil, &block) + observe(authorization_context:) do |event| + block.call(event) if event.fetch("name") == "solid_objects.#{name}" + end + end + # @rbs (?available_at: Time?, ?idempotency_key: String?, ?authorization_context: untyped) -> OperationDispatcher def async(available_at: nil, idempotency_key: nil, authorization_context: nil) OperationDispatcher.new( diff --git a/lib/solid_objects/reminder_scheduler.rb b/lib/solid_objects/reminder_scheduler.rb index 5236c1c..4e635de 100644 --- a/lib/solid_objects/reminder_scheduler.rb +++ b/lib/solid_objects/reminder_scheduler.rb @@ -166,6 +166,10 @@ def enqueue(reminder, now:) mailbox.announce(message) SolidObjects.instrument( :"reminder.enqueued", + instance_id: reminder.instance_id, + message_id: message.id, + attempt: message.attempt_count, + lateness_milliseconds: [ (((now || database_adapter.database_now) - reminder.next_run_at) * 1000).round, 0 ].max, reminder_id: reminder.id, actor_type: reminder.actor_type, actor_id: reminder.actor_id, diff --git a/lib/solid_objects/state_snapshot.rb b/lib/solid_objects/state_snapshot.rb index 6593020..aadd034 100644 --- a/lib/solid_objects/state_snapshot.rb +++ b/lib/solid_objects/state_snapshot.rb @@ -10,6 +10,7 @@ def initialize(reference) actor_snapshot = ActorSnapshot.new(reference) @actor_class = actor_snapshot.actor_class @data = Serialization.readonly_copy(actor_snapshot.actor.state.to_h) + SolidObjects.instrument(:"snapshot.read", actor_type: reference.actor_type, actor_id: reference.actor_id, instance_id: actor_snapshot.instance_id.zero? ? nil : actor_snapshot.instance_id, revision: actor_snapshot.revision) end # @rbs () -> Hash[String, untyped] diff --git a/lib/solid_objects/supervisor.rb b/lib/solid_objects/supervisor.rb index f3f27e3..84bed8d 100644 --- a/lib/solid_objects/supervisor.rb +++ b/lib/solid_objects/supervisor.rb @@ -97,8 +97,7 @@ def monitor_loop rescue => error SolidObjects.instrument( :"supervisor.monitor_failed", - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) end sleep SolidObjects.configuration.supervisor_monitor_interval @@ -161,8 +160,7 @@ def retention_loop failures += 1 SolidObjects.instrument( :"supervisor.retention_failed", - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) end wait_for_next_retention(failures) @@ -220,8 +218,7 @@ def advance_redrive rescue => error SolidObjects.instrument( :"supervisor.redrive_failed", - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) false end diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb new file mode 100644 index 0000000..f705a99 --- /dev/null +++ b/lib/solid_objects/telemetry.rb @@ -0,0 +1,73 @@ +# rbs_inline: enabled + +module SolidObjects + module Telemetry + FIELDS = %w[ + actorType actorId instanceId incarnation revision messageId requestId attempt sequence + operation deliveryMode generation durationMilliseconds latenessMilliseconds ageMilliseconds + depth count errorName outcome retryable status effectId effectName reminderId occurrence + truncated broadcastId code role reason processId processKind ownerId componentCount byteCount + thresholdBytes previousRunAt nextRunAt name commitAction payload + ].freeze + ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds" }.freeze + + class << self + # @rbs (Symbol, Hash[Symbol, untyped]) -> void + def emit(name, payload) + observer = SolidObjects.configuration.instrumentation + return unless observer + + observer.call(event(name, payload)) + rescue + nil + end + + # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] + def event(name, payload) + attributes = safe_attributes(payload) + adapter = DatabaseAdapter.family(Record.connection).to_s + event_name = "solid_objects.#{name}" + labels = { "event" => event_name, "adapter" => adapter, "actorType" => attributes.fetch("actorType", "") } + metrics = [ { "name" => "solid_objects.events", "kind" => "counter", "unit" => "1", "value" => 1, "labels" => labels } ] + [ + [ "durationMilliseconds", "solid_objects.duration", "histogram", "ms" ], + [ "latenessMilliseconds", "solid_objects.reminder.lateness", "histogram", "ms" ], + [ "ageMilliseconds", "solid_objects.outbox.age", "histogram", "ms" ], + [ "depth", "solid_objects.mailbox.depth", "gauge", "1" ] + ].each do |field, metric, kind, unit| + value = attributes[field] + next unless value.is_a?(Numeric) && value.finite? + + metrics << { "name" => metric, "kind" => kind, "unit" => unit, "value" => [ value, 0 ].max, "labels" => labels } + end + Serialization.readonly_copy( + "schemaVersion" => 1, + "name" => event_name, + "occurredAt" => Time.now.utc.iso8601(3), + "adapter" => adapter, + "actorType" => attributes["actorType"]&.to_s, + "actorId" => attributes["actorId"]&.to_s, + "incarnation" => (attributes["incarnation"] || attributes["instanceId"])&.to_s, + "revision" => attributes["revision"]&.to_s, + "messageId" => attributes["messageId"]&.to_s, + "attempt" => attributes.fetch("attempt", 0), + "attributes" => attributes, + "metrics" => metrics + ) + end + + # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] + def safe_attributes(payload) + payload.each_with_object({}) do |(key, value), attributes| + name = key.to_s.camelize(:lower) + name = ALIASES.fetch(name, name) + next unless FIELDS.include?(name) + next unless value.nil? || value.is_a?(String) || value.is_a?(Numeric) || value == true || value == false + + value = value.to_s if !value.nil? && (name.end_with?("Id") || %w[revision sequence generation].include?(name)) + attributes[name] = value + end + end + end + end +end diff --git a/lib/solid_objects/wake_up_adapters/postgresql.rb b/lib/solid_objects/wake_up_adapters/postgresql.rb index 8c45271..501f5fe 100644 --- a/lib/solid_objects/wake_up_adapters/postgresql.rb +++ b/lib/solid_objects/wake_up_adapters/postgresql.rb @@ -145,8 +145,7 @@ def instrument_failure(operation, error) :"wake_up.failed", adapter: "postgresql", operation: operation.to_s, - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) end end diff --git a/lib/solid_objects/wake_up_adapters/redis.rb b/lib/solid_objects/wake_up_adapters/redis.rb index bc44dc5..beae8f1 100644 --- a/lib/solid_objects/wake_up_adapters/redis.rb +++ b/lib/solid_objects/wake_up_adapters/redis.rb @@ -208,8 +208,7 @@ def instrument_failure(operation, error) :"wake_up.failed", adapter: "redis", operation: operation.to_s, - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) end end diff --git a/lib/solid_objects/worker.rb b/lib/solid_objects/worker.rb index 888aefc..f83ae39 100644 --- a/lib/solid_objects/worker.rb +++ b/lib/solid_objects/worker.rb @@ -69,8 +69,7 @@ def run_once SolidObjects.configuration.logger.error( event: "solid_objects.worker.error", process_id: process_registry.process_record&.id, - error_class: error.class.name, - error_message: error.message + error_class: error.class.name ) 0 end diff --git a/sig/generated/lib/solid_objects/actor_channel.rbs b/sig/generated/lib/solid_objects/actor_channel.rbs index e6d45a2..f034f23 100644 --- a/sig/generated/lib/solid_objects/actor_channel.rbs +++ b/sig/generated/lib/solid_objects/actor_channel.rbs @@ -7,6 +7,9 @@ module SolidObjects # @rbs () -> void def subscribed: () -> void + # @rbs () -> void + def unsubscribed: () -> void + private attr_reader reference: untyped diff --git a/sig/generated/lib/solid_objects/configuration.rbs b/sig/generated/lib/solid_objects/configuration.rbs index cd1c4ca..fb9989e 100644 --- a/sig/generated/lib/solid_objects/configuration.rbs +++ b/sig/generated/lib/solid_objects/configuration.rbs @@ -4,6 +4,8 @@ module SolidObjects class Configuration @transmission_actor_type_resolver: Proc + @reminder_scheduler_count: Integer + @broadcast_worker_count: Integer @effect_worker_count: Integer @@ -34,8 +36,6 @@ module SolidObjects @supervisor_monitor_interval: Float - @table_name_prefix: String - @administration_identity: Proc @authorize_transmission: Proc @@ -64,9 +64,11 @@ module SolidObjects @logger: untyped + @instrumentation: Proc? + @connects_to: Hash[Symbol, untyped]? - @reminder_scheduler_count: Integer + @table_name_prefix: String @polling_interval: Float @@ -186,6 +188,8 @@ module SolidObjects attr_accessor logger: untyped + attr_accessor instrumentation: untyped + attr_accessor stream_signing_secret: untyped attr_accessor broadcast_adapter: untyped diff --git a/sig/generated/lib/solid_objects/diagnostics.rbs b/sig/generated/lib/solid_objects/diagnostics.rbs new file mode 100644 index 0000000..fea1ec8 --- /dev/null +++ b/sig/generated/lib/solid_objects/diagnostics.rbs @@ -0,0 +1,25 @@ +# Generated from lib/solid_objects/diagnostics.rb with RBS::Inline + +module SolidObjects + class Diagnostics + # @rbs @reference: Reference + # @rbs (Reference) -> void + def initialize: (Reference) -> void + + # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + def summary: (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + + # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def observe: (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + + private + + attr_reader reference: untyped + + # @rbs (Symbol, ?authorization_context: untyped) -> void + def authorize!: (Symbol, ?authorization_context: untyped) -> void + + # @rbs (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] + def summarize: (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] + end +end diff --git a/sig/generated/lib/solid_objects/executor.rbs b/sig/generated/lib/solid_objects/executor.rbs index 1733c1a..cda9890 100644 --- a/sig/generated/lib/solid_objects/executor.rbs +++ b/sig/generated/lib/solid_objects/executor.rbs @@ -4,6 +4,8 @@ module SolidObjects class Executor @completion_transaction: untyped + @started_at: Float + @message: Message @activation: Activation @@ -103,6 +105,9 @@ module SolidObjects # @rbs (message: Message, error_details: Hash[String, untyped], now: Time) -> DeadLetter def create_dead_letter: (message: Message, error_details: Hash[String, untyped], now: Time) -> DeadLetter + # @rbs () -> bool + def recovery_message?: () -> bool + # @rbs () -> Hash[Symbol, untyped] def instrumentation_payload: () -> Hash[Symbol, untyped] end diff --git a/sig/generated/lib/solid_objects/reference.rbs b/sig/generated/lib/solid_objects/reference.rbs index fa555e1..834ea37 100644 --- a/sig/generated/lib/solid_objects/reference.rbs +++ b/sig/generated/lib/solid_objects/reference.rbs @@ -13,6 +13,15 @@ module SolidObjects # @rbs (actor_type: String, actor_id: String) -> void def initialize: (actor_type: String, actor_id: String) -> void + # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + def diagnostics: (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + + # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def observe: (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + + # @rbs (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + def on: (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (?available_at: Time?, ?idempotency_key: String?, ?authorization_context: untyped) -> OperationDispatcher def async: (?available_at: Time?, ?idempotency_key: String?, ?authorization_context: untyped) -> OperationDispatcher diff --git a/sig/generated/lib/solid_objects/telemetry.rbs b/sig/generated/lib/solid_objects/telemetry.rbs new file mode 100644 index 0000000..2629ece --- /dev/null +++ b/sig/generated/lib/solid_objects/telemetry.rbs @@ -0,0 +1,18 @@ +# Generated from lib/solid_objects/telemetry.rb with RBS::Inline + +module SolidObjects + module Telemetry + FIELDS: untyped + + ALIASES: untyped + + # @rbs (Symbol, Hash[Symbol, untyped]) -> void + def self.emit: (Symbol, Hash[Symbol, untyped]) -> void + + # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] + def self.event: (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] + + # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] + def self.safe_attributes: (Hash[Symbol, untyped]) -> Hash[String, untyped] + end +end diff --git a/test/integration/actor_channel_test.rb b/test/integration/actor_channel_test.rb index fc9ff92..68843b2 100644 --- a/test/integration/actor_channel_test.rb +++ b/test/integration/actor_channel_test.rb @@ -68,6 +68,8 @@ def stream_from(_broadcasting, callback = nil, coder: nil, &block) end test "subscribes to scalar updates through rendered Turbo data" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } reference = ChannelActor.ref("actor-1") SolidObjects.configuration.authorize_subscription = ->(**) { true } parameters = rendered_subscription_parameters(reference) do |actor| @@ -96,6 +98,9 @@ def stream_from(_broadcasting, callback = nil, coder: nil, &block) updates = transmissions.select { |transmission| transmission.include?(target) } assert_equal 2, updates.length assert_includes updates.last, ">1" + unsubscribe + assert_includes events.map { |event| event.fetch("name") }, "solid_objects.realtime.connected" + assert_includes events.map { |event| event.fetch("name") }, "solid_objects.realtime.disconnected" ensure worker&.stop end diff --git a/test/integration/actor_code_write_guard_test.rb b/test/integration/actor_code_write_guard_test.rb index 0885c56..122aeda 100644 --- a/test/integration/actor_code_write_guard_test.rb +++ b/test/integration/actor_code_write_guard_test.rb @@ -68,10 +68,13 @@ def run end test "activation hook writes fail before actor code runs" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } error = assert_raises(SolidObjects::ApplicationWriteForbidden) do ActivatingActor.ref("one").run end + assert events.any? { |event| event.fetch("name") == "solid_objects.activation.failed" } assert_equal "on_activate", error.operation assert_empty SolidObjectsTestDomainRecord.all assert_equal "ready", SolidObjects::MessageReference.from_message(SolidObjects::Message.last).status diff --git a/test/integration/process_lifecycle_test.rb b/test/integration/process_lifecycle_test.rb index d7cacae..a40fa6e 100644 --- a/test/integration/process_lifecycle_test.rb +++ b/test/integration/process_lifecycle_test.rb @@ -14,6 +14,8 @@ def run end test "recovers a claimed message after its worker heartbeat expires" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } message_reference = RecoveryActor.ref("one").async.run message = SolidObjects::Message.find(message_reference.id) instance = message.instance @@ -41,6 +43,7 @@ def run worker.run_until_idle assert_equal({ "runs" => 1 }, instance.reload.state) + assert events.any? { |event| event.fetch("name") == "solid_objects.recovery.reclaimed" && event.fetch("attempt") == 2 } assert_equal 2, message.reload.attempt_count assert message.completed? ensure diff --git a/test/integration/telemetry_test.rb b/test/integration/telemetry_test.rb new file mode 100644 index 0000000..ad42256 --- /dev/null +++ b/test/integration/telemetry_test.rb @@ -0,0 +1,121 @@ +# frozen_string_literal: true + +require "database_test_helper" + +class TelemetryTest < ActiveSupport::TestCase + class Counter < SolidObjects::Actor + actor_type "telemetry-counter" + attribute :count, default: 0 + + def arrange + emit :telemetry_effect, secret: "private effect" + schedule(at: 1.minute.ago).increment + end + + def fail_operation + raise "private failure" + end + + def increment + self.count += 1 + end + end + + test "a failing started subscriber cannot fail a turn" do + subscriber = ActiveSupport::Notifications.subscribe("solid_objects.message.started") { raise "private sink failure" } + assert_equal 1, Counter.ref("one").increment + assert_equal 1, Counter.ref("one").snapshot.count + ensure + ActiveSupport::Notifications.unsubscribe(subscriber) + end + + test "portable events carry correlation and bounded metric labels" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + Counter.ref("one").increment + event = events.find { |entry| entry.fetch("name") == "solid_objects.message.completed" } + assert_equal 1, event.fetch("schemaVersion") + assert_equal database_family.to_s, event.fetch("adapter") + assert_equal "one", event.fetch("actorId") + assert_equal 1, event.fetch("attempt") + assert event.fetch("incarnation") + refute_includes event.fetch("metrics").to_json, "actorId" + end + test "diagnostics and observers require authorization and are bounded" do + reference = Counter.ref("diagnostics") + assert_raises(SolidObjects::Unauthorized) { reference.diagnostics } + assert_raises(SolidObjects::Unauthorized) { reference.observe { nil } } + SolidObjects.configuration.authorize_administration = ->(authorization_context:, **) { authorization_context == "operator" } + events = [] + stop = reference.on("message.enqueued", authorization_context: "operator") { |event| events << event } + 2.times { reference.async.increment } + Counter.ref("other").async.increment + assert_equal 2, events.length + stop.call + reference.async.increment + assert_equal 2, events.length + summary = reference.diagnostics(authorization_context: "operator", limit: 1) + assert_equal 1, summary.fetch("mailbox").fetch("sampled") + assert summary.fetch("mailbox").fetch("truncated") + assert summary.fetch("mailbox").frozen? + assert_raises(ArgumentError) { reference.diagnostics(authorization_context: "operator", limit: 101) } + end + + test "events cover retry failure snapshot reminder and outbox without private data" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + SolidObjects.configuration.max_attempts = 2 + SolidObjects.configuration.retry_delay = ->(_) { 0 } + SolidObjects.configuration.authorize_administration = ->(**) { true } + SolidObjects.register_effect(:telemetry_effect) { "private provider response" } + reference = Counter.ref("lifecycle") + reference.async.fail_operation + worker = SolidObjects::Worker.new + worker.run_once + reference.async.arrange + worker.run_once + summary = reference.diagnostics + assert_equal 1, summary.fetch("outbox").fetch("sampled") + assert_equal 1, summary.fetch("reminders").fetch("sampled") + effect_executor = SolidObjects::EffectExecutor.new + effect_executor.run_once + scheduler = SolidObjects::ReminderScheduler.new + scheduler.run_once + reference.snapshot + names = events.map { |event| event.fetch("name") } + %w[activation.started activation.completed message.retry dead_letter.created reminder.enqueued outbox.age snapshot.read].each do |name| + assert_includes names, "solid_objects.#{name}" + end + refute_includes events.to_json, "private" + ensure + worker&.stop + effect_executor&.stop + scheduler&.stop + end + + test "portable observers exclude unknown attributes and cannot mask application errors" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + error = assert_raises(RuntimeError) do + SolidObjects.instrument(:custom, arguments: "private", state: { secret: "private" }, password: "private") { raise "application failure" } + end + assert_equal "application failure", error.message + assert_empty events.last.fetch("attributes") + SolidObjects.configuration.instrumentation = ->(_) { raise "exporter failure" } + assert_equal 1, Counter.ref("safe").increment + end + test "reports failed durable recovery callbacks in diagnostics" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + SolidObjects.configuration.authorize_administration = ->(**) { true } + SolidObjects.configuration.max_attempts = 1 + reference = Counter.ref("recovery") + SolidObjects::Mailbox.new.enqueue(reference:, operation: "fail_operation", arguments: {}, delivery_mode: "internal", idempotency_key: "effect:test:recovery") + worker = SolidObjects::Worker.new + worker.run_once + assert_equal 1, reference.diagnostics.fetch("recoveryFailures").fetch("sampled") + assert events.any? { |event| event.fetch("name") == "solid_objects.recovery.failed" } + ensure + worker&.stop + end +end From 7149e9ca614a7b162bcc96db002191015cc822aa Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 08:44:42 -0700 Subject: [PATCH 02/11] fix: Isolate outbox telemetry measurements Keep optional database measurements from failing durable delivery. Cover broadcast age and report message duration only on terminal outcomes so Ruby and JavaScript emit matching samples. --- docs/observability.md | 3 ++ lib/solid_objects/broadcast_executor.rb | 1 + lib/solid_objects/effect_executor.rb | 2 +- lib/solid_objects/executor.rb | 14 ++++-- lib/solid_objects/telemetry.rb | 23 +++++++++- sig/generated/lib/solid_objects/executor.rbs | 3 ++ sig/generated/lib/solid_objects/telemetry.rbs | 3 ++ test/integration/telemetry_test.rb | 43 +++++++++++++++++++ 8 files changed, 85 insertions(+), 7 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 061c1aa..e31f2c2 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -92,6 +92,9 @@ confers no permission. Diagnostics read at most `limit + 1` rows per queue source, with a hard limit of 100. Each category returns `sampled`, `truncated`, and `oldestAgeMilliseconds`. +The limit applies to each combined category: one effect plus one broadcast with +`limit: 1` returns `sampled: 1, truncated: true`, even when both source queries +returned all their rows. The extra row proves that the category exceeds its cap. The last value measures nonnegative time since availability (or terminal failure for recovery callbacks); future reminders have zero age. No payloads or row identifiers are returned. Samples are observations across several queries, diff --git a/lib/solid_objects/broadcast_executor.rb b/lib/solid_objects/broadcast_executor.rb index 1aaa574..44584fc 100644 --- a/lib/solid_objects/broadcast_executor.rb +++ b/lib/solid_objects/broadcast_executor.rb @@ -41,6 +41,7 @@ def run_once broadcast = claim_next return false unless broadcast + Telemetry.outbox(broadcast) broadcast_adapter.call(broadcast) complete(broadcast) true diff --git a/lib/solid_objects/effect_executor.rb b/lib/solid_objects/effect_executor.rb index cdf83d3..00be8da 100644 --- a/lib/solid_objects/effect_executor.rb +++ b/lib/solid_objects/effect_executor.rb @@ -137,7 +137,7 @@ def claim_next # @rbs (Effect) -> untyped def deliver(effect) - SolidObjects.instrument(:"outbox.age", instance_id: effect.instance_id, actor_type: effect.instance.actor_type, actor_id: effect.instance.actor_id, message_id: effect.message_id, attempt: effect.attempt_count, age_milliseconds: [ ((database_adapter.database_now - effect.available_at) * 1000).round, 0 ].max) + Telemetry.outbox(effect) return deliver_actor_message(effect) if effect.name == ACTOR_MESSAGE_EFFECT handler = SolidObjects.effect_registry.fetch(effect.name) diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index d05609e..c397ddb 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -161,7 +161,7 @@ def complete(result, observable_changes, state_after:, state_changed:) end SolidObjects.instrument_after_commit(:"recovery.completed", **instrumentation_payload) if recovery_message? report_large_state(state_after.byte_size) - SolidObjects.instrument_after_commit(:"message.completed", **instrumentation_payload, revision: message.sequence) + SolidObjects.instrument_after_commit(:"message.completed", **instrumentation_payload, revision: message.sequence, duration_milliseconds: elapsed_milliseconds) SolidObjects.wake_up.signal rescue CommittedTransactionError raise @@ -424,6 +424,7 @@ def fail_message(error) :"message.failed", **instrumentation_payload, error_class: error.class.name, + duration_milliseconds: elapsed_milliseconds, dead: ) SolidObjects.instrument_after_commit(:"recovery.failed", **instrumentation_payload) if dead && recovery_message? @@ -463,7 +464,8 @@ def reject_message(rejection) SolidObjects.instrument_after_commit( :"message.rejected", **instrumentation_payload, - code: rejection.code + code: rejection.code, + duration_milliseconds: elapsed_milliseconds ) SolidObjects.wake_up.signal end @@ -530,6 +532,11 @@ def recovery_message? message.delivery_mode == "internal" && message.idempotency_key.to_s.start_with?("effect:") && message.idempotency_key.to_s.end_with?(":recovery") end + # @rbs () -> (Integer | Float) + def elapsed_milliseconds + ((::Process.clock_gettime(::Process::CLOCK_MONOTONIC) - @started_at) * 1000).round(3) + end + # @rbs () -> Hash[Symbol, untyped] def instrumentation_payload { @@ -539,8 +546,7 @@ def instrumentation_payload actor_id: message.actor_id, sequence: message.sequence, attempt: message.attempt_count, - request_id: message.request_id, - duration_milliseconds: ((::Process.clock_gettime(::Process::CLOCK_MONOTONIC) - @started_at) * 1000).round(3) + request_id: message.request_id } end end diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb index f705a99..310c527 100644 --- a/lib/solid_objects/telemetry.rb +++ b/lib/solid_objects/telemetry.rb @@ -6,12 +6,31 @@ module Telemetry actorType actorId instanceId incarnation revision messageId requestId attempt sequence operation deliveryMode generation durationMilliseconds latenessMilliseconds ageMilliseconds depth count errorName outcome retryable status effectId effectName reminderId occurrence - truncated broadcastId code role reason processId processKind ownerId componentCount byteCount + outboxKind truncated broadcastId code role reason processId processKind ownerId componentCount byteCount thresholdBytes previousRunAt nextRunAt name commitAction payload ].freeze ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds" }.freeze class << self + # @rbs (Effect | Broadcast) -> void + def outbox(record) + return unless SolidObjects.configuration.instrumentation || ActiveSupport::Notifications.notifier.listening?("solid_objects.outbox.age") + + instance = record.instance + SolidObjects.instrument( + :"outbox.age", + instance_id: record.instance_id, + actor_type: instance.actor_type, + actor_id: instance.actor_id, + message_id: record.message_id, + attempt: record.attempt_count, + outbox_kind: record.is_a?(Effect) ? "effect" : "broadcast", + age_milliseconds: [ ((SolidObjects.database_adapter.database_now - record.available_at) * 1000).round, 0 ].max + ) + rescue + nil + end + # @rbs (Symbol, Hash[Symbol, untyped]) -> void def emit(name, payload) observer = SolidObjects.configuration.instrumentation @@ -27,7 +46,7 @@ def event(name, payload) attributes = safe_attributes(payload) adapter = DatabaseAdapter.family(Record.connection).to_s event_name = "solid_objects.#{name}" - labels = { "event" => event_name, "adapter" => adapter, "actorType" => attributes.fetch("actorType", "") } + labels = { "event" => event_name, "adapter" => adapter, "actorType" => attributes["actorType"].to_s } metrics = [ { "name" => "solid_objects.events", "kind" => "counter", "unit" => "1", "value" => 1, "labels" => labels } ] [ [ "durationMilliseconds", "solid_objects.duration", "histogram", "ms" ], diff --git a/sig/generated/lib/solid_objects/executor.rbs b/sig/generated/lib/solid_objects/executor.rbs index cda9890..95f7bd9 100644 --- a/sig/generated/lib/solid_objects/executor.rbs +++ b/sig/generated/lib/solid_objects/executor.rbs @@ -108,6 +108,9 @@ module SolidObjects # @rbs () -> bool def recovery_message?: () -> bool + # @rbs () -> (Integer | Float) + def elapsed_milliseconds: () -> (Integer | Float) + # @rbs () -> Hash[Symbol, untyped] def instrumentation_payload: () -> Hash[Symbol, untyped] end diff --git a/sig/generated/lib/solid_objects/telemetry.rbs b/sig/generated/lib/solid_objects/telemetry.rbs index 2629ece..08e4b61 100644 --- a/sig/generated/lib/solid_objects/telemetry.rbs +++ b/sig/generated/lib/solid_objects/telemetry.rbs @@ -6,6 +6,9 @@ module SolidObjects ALIASES: untyped + # @rbs (Effect | Broadcast) -> void + def self.outbox: (Effect | Broadcast) -> void + # @rbs (Symbol, Hash[Symbol, untyped]) -> void def self.emit: (Symbol, Hash[Symbol, untyped]) -> void diff --git a/test/integration/telemetry_test.rb b/test/integration/telemetry_test.rb index ad42256..8e0e537 100644 --- a/test/integration/telemetry_test.rb +++ b/test/integration/telemetry_test.rb @@ -6,6 +6,7 @@ class TelemetryTest < ActiveSupport::TestCase class Counter < SolidObjects::Actor actor_type "telemetry-counter" attribute :count, default: 0 + observable :count, broadcast: :value def arrange emit :telemetry_effect, secret: "private effect" @@ -40,6 +41,8 @@ def increment assert_equal 1, event.fetch("attempt") assert event.fetch("incarnation") refute_includes event.fetch("metrics").to_json, "actorId" + started = events.find { |entry| entry.fetch("name") == "solid_objects.message.started" } + refute_includes started.fetch("metrics").map { |metric| metric.fetch("name") }, "solid_objects.duration" end test "diagnostics and observers require authorization and are bounded" do reference = Counter.ref("diagnostics") @@ -118,4 +121,44 @@ def increment ensure worker&.stop end + test "outbox measurement failure cannot fail delivery" do + SolidObjects.configuration.instrumentation = ->(_) { true } + Counter.ref("measurement").arrange + calls = [] + SolidObjects.register_effect(:telemetry_effect) { + calls << :delivered + "result" + } + executor = SolidObjects::EffectExecutor.new + executor.define_singleton_method(:claim_next) do + super().tap do |effect| + effect.define_singleton_method(:available_at) { raise "measurement failed" } + end + end + assert executor.run_once + assert_equal [ :delivered ], calls + assert_equal "completed", SolidObjects::Effect.first.status + ensure + executor&.stop + end + + test "samples broadcast age and caps the combined outbox category" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + SolidObjects.configuration.authorize_administration = ->(**) { true } + SolidObjects.configuration.broadcast_adapter = ->(_) { true } + reference = Counter.ref("broadcast") + reference.increment + reference.arrange + summary = reference.diagnostics(limit: 1) + assert_equal 1, summary.fetch("outbox").fetch("sampled") + assert summary.fetch("outbox").fetch("truncated") + executor = SolidObjects::BroadcastExecutor.new + assert executor.run_once + event = events.find { |entry| entry.fetch("name") == "solid_objects.outbox.age" && entry.fetch("attributes")["outboxKind"] == "broadcast" } + assert event + assert_operator event.fetch("metrics").last.fetch("value"), :>=, 0 + ensure + executor&.stop + end end From 01b298194db4fae9abd066aaa2204a5b14394187 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 09:39:20 -0700 Subject: [PATCH 03/11] fix: Align message results and authorization Reauthorize message reads and retain bounded JSON results for background operations so Ruby and JavaScript provide the same outcome contract. Normalize polling telemetry units and preserve the matching transmit validation and ordering guarantees in shared fixtures and documentation. --- CHANGELOG.md | 10 +++ app/models/solid_objects/message.rb | 15 ++++ compatibility/transmit-envelopes.json | 10 +++ docs/architecture.md | 12 +++- docs/authorization.md | 6 +- docs/observability.md | 4 ++ docs/roadmap.md | 3 + docs/security.md | 4 +- docs/transmission.md | 5 +- lib/solid_objects/client.rb | 13 ++++ lib/solid_objects/executor.rb | 2 +- lib/solid_objects/message_reference.rb | 18 ++--- lib/solid_objects/synchronous_invocation.rb | 22 +----- lib/solid_objects/telemetry.rb | 6 ++ sig/generated/lib/solid_objects/client.rbs | 3 + .../lib/solid_objects/message_reference.rbs | 12 ++-- .../solid_objects/synchronous_invocation.rbs | 3 - .../models/solid_objects/message.rbs | 3 + test/integration/result_lookup_test.rb | 72 ++++++++++++++++++- test/integration/retry_test.rb | 2 +- .../synchronous_invocation_test.rb | 3 +- test/integration/telemetry_test.rb | 8 +++ test/integration/transmit_staging_test.rb | 21 ++++++ 23 files changed, 206 insertions(+), 51 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a6e90f..bd87df4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +- Reauthorize every message-reference status, result, and outcome read against + the original invocation. Pass `authorization_context:` on every read. +- Retain immutable JSON results for background and internal messages as well as + synchronous calls. All operations now enforce result serialization and size + limits; return `nil` explicitly when an operation does not need a result. + `result` raises terminal rejection/failure errors; `outcome` exposes them as data. +- Preserve polling transition intervals in milliseconds and string reasons in + portable telemetry. Pin transmit staging order and null-argument validation + against the shared JavaScript contract. + - Add portable telemetry, isolated observer hooks, metric definitions, and bounded authorized actor diagnostics matching JavaScript. diff --git a/app/models/solid_objects/message.rb b/app/models/solid_objects/message.rb index d44285b..5ab5a32 100644 --- a/app/models/solid_objects/message.rb +++ b/app/models/solid_objects/message.rb @@ -47,6 +47,21 @@ def dead? dead_letter.present? end + # @rbs () -> untyped + def result! + if rejected? + raise Rejected.new( + code: rejection.fetch("code"), + message: rejection.fetch("message"), + details: rejection.fetch("details"), + message_id: id + ) + end + raise MessageFailed.new("actor message failed permanently", message_id: id, details: error || {}) if dead? + + Serialization.readonly_copy(result) + end + private # @rbs () -> void diff --git a/compatibility/transmit-envelopes.json b/compatibility/transmit-envelopes.json index 93d3459..a1b9249 100644 --- a/compatibility/transmit-envelopes.json +++ b/compatibility/transmit-envelopes.json @@ -84,6 +84,16 @@ "operation": "increment", "arguments": [ 1 ] } + }, + { + "name": "null arguments", + "envelope": { + "effectId": "fixture-effect-0008", + "actorType": "transmit-counters", + "actorId": "fixture-counter", + "operation": "increment", + "arguments": null + } } ] } diff --git a/docs/architecture.md b/docs/architecture.md index 845a388..5a40d02 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -431,8 +431,16 @@ Each lookup runs the authorization hook the original call ran, against the stored operation and arguments, and answers `nil` for an absent row, an unregistered actor, and a refused caller alike, so it cannot be used to ask whether a request id exists. `MessageReference#outcome` reports the status, the -result, the persisted error, the rejection, and the attempt count. A result is -stored for `sync` delivery only. +result, the persisted error, the rejection, and the attempt count. Every delivery +mode stores its JSON result, including background messages. Result serialization +and `max_result_bytes` apply before commit; a result that cannot be stored fails +the turn. Return `nil` explicitly from operations that need no result. + +`status`, `result`, and `outcome` reauthorize the stored operation on every read. +Pass `authorization_context:` each time; references retain identity rather than +caller permissions. `result` raises `Rejected` or `MessageFailed` for terminal +errors and returns a deeply frozen successful value. `outcome` reports terminal +errors as data. An actor remembers the idempotency keys of its own finished turns. The executor already writes the instance row in the transaction that completes, rejects, or diff --git a/docs/authorization.md b/docs/authorization.md index 461dc27..468bea8 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -15,8 +15,10 @@ answers nothing until the host application defines its trust boundary. | `authorize_subscription` | Action Cable subscription to one actor stream | The `ActionCable::Connection` object | Clients can receive future observable updates for other actors | | `authorize_administration` | Engine administration controllers, every `SolidObjects::Web` page, process inspection/cleanup/pruning, message pruning, and dead-letter inspection/retry | Rails controller, a `SolidObjects::Web` request that answers `request`/`session`/`env`, or `{ source: "cli" }` | Operational metadata, arguments, errors, deletion, and retries become exposed or mutable | -Waiting again through `MessageReference#wait` reauthorizes the stored -invocation as a message or query. Internal reminder, effect-callback, and +Every `MessageReference#status`, `#result`, `#outcome`, and `#wait` call +reauthorizes the stored operation and arguments as a message or query. Supply +`authorization_context:` on each call, including after `find_by`; the reference +does not retain the original caller's context. Internal reminder, effect-callback, and actor-to-actor deliveries come from already committed runtime rows and do not re-enter the public client policy. diff --git a/docs/observability.md b/docs/observability.md index e31f2c2..ff19d2d 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -43,6 +43,10 @@ Events describe local observations. Concurrent deletion, crashes, and failed exporters can omit events. Never infer exactly-once delivery from event counts. Additional existing runtime events retain their names. +Portable `polling.interval_changed` events carry `previousIntervalMilliseconds`, +`currentIntervalMilliseconds`, and a string `reason`. Ruby converts its native +second-based notification values while preserving the original notification. + ## Metrics and tracing Metrics are sample descriptions. Exporting them is opt-in: the runtime does not diff --git a/docs/roadmap.md b/docs/roadmap.md index 06b10ac..b77682c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -122,6 +122,9 @@ batched and unbatched components, an inert replay of an applied revision, cancellation of the request left in flight by the drop, incarnation ordering after a destroy and recreate, and payload delivery exactly once per revision +- Authorized message-reference reads recheck the original operation and arguments, + return immutable JSON results for every delivery mode, and raise terminal errors + from `result` while `outcome` exposes them as data. - Result lookup by request ID and by idempotency key, authorized with the hook the original call ran and against the stored operation and arguments. An actor remembers the idempotency keys of its own last `retained_idempotency_keys` diff --git a/docs/security.md b/docs/security.md index 9c941bb..09478ec 100644 --- a/docs/security.md +++ b/docs/security.md @@ -16,8 +16,8 @@ delegate to the authorized synchronous invocation path. Keep implementation helpers private or protected. Query, attribute, observable, and committed `snapshot` reads use the separate query authorization policy. Explicit `async` message delivery uses the same message authorization policy as direct calls. -Recovering a timed-out result through `MessageReference#wait` reauthorizes the -stored operation. +Message reference status, result, outcome, and wait reads reauthorize the stored +operation and arguments with the context supplied to that read. `reference.destroy` delegates to `authorize_destroy` before checking whether the actor exists, so denial does not reveal actor existence. diff --git a/docs/transmission.md b/docs/transmission.md index dd205fe..b199f3e 100644 --- a/docs/transmission.md +++ b/docs/transmission.md @@ -18,6 +18,9 @@ actor does the same with `transmit.increment(amount:)`. Either ingest accepts either sender, so Rails-to-Rails, Rails-to-Node, Node-to-Rails, and browser-to-Rails replication all ride one contract. +An omitted `arguments` field defaults to `{}`. Explicit `null` and arrays are +rejected by both runtimes. Shared fixtures cover this distinction. + ## The sending side ```ruby @@ -57,7 +60,7 @@ retries with backoff and dead-letters on exhaustion, like any other effect. The drain keeps per-actor order across failures: a claimed transmit effect delivers every undelivered sibling for its actor up to its own mailbox -sequence, oldest first. The receiving side dedups on `transmit:`, +sequence, oldest first, preserving staging order within each turn. The receiving side dedups on `transmit:`, so a redelivered envelope applies once. Delivery is at-least-once by design, and the drain accepts redundant sends diff --git a/lib/solid_objects/client.rb b/lib/solid_objects/client.rb index 20c4936..c15d90d 100644 --- a/lib/solid_objects/client.rb +++ b/lib/solid_objects/client.rb @@ -94,6 +94,19 @@ def wait(message_reference, timeout:, authorization_context: nil) raise SyncDiagnostics.new.database_contention_for(message_reference, timeout:) end + # @rbs (MessageReference, ?authorization_context: untyped) -> Message + def read_message(message_reference, authorization_context: nil) + Message.uncached do + message = Message.includes(:ready_message, :claimed_message, :dead_letter).find(message_reference.id) + validate_message_reference!(message_reference, message) + raise Unauthorized, "message result is not authorized" unless authorized_to_read?(message, authorization_context:) + + message + end + rescue ActiveRecord::RecordNotFound, ActorDestroyed + raise Unauthorized, "message result is not authorized" + end + # @rbs (?reference: Reference?, ?request_id: String?, ?idempotency_key: String?, ?authorization_context: untyped) -> MessageReference? def find_by(reference: nil, request_id: nil, idempotency_key: nil, authorization_context: nil) unless [ request_id, idempotency_key ].compact.one? diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index c397ddb..d5cf11b 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -89,7 +89,7 @@ def changed_observables(before, after) def complete(result, observable_changes, state_after:, state_changed:) ensure_state_fits!(state_after.byte_size) serialized_result = Serialization.dump( - (message.delivery_mode == "sync") ? result : nil, + result, max_bytes: SolidObjects.configuration.max_result_bytes ) effect_intents = actor.drain_effect_intents diff --git a/lib/solid_objects/message_reference.rb b/lib/solid_objects/message_reference.rb index afed7c1..ade6048 100644 --- a/lib/solid_objects/message_reference.rb +++ b/lib/solid_objects/message_reference.rb @@ -33,20 +33,20 @@ def initialize(id:, request_id:, actor_type:, actor_id:, sequence:) freeze end - # @rbs () -> String - def status - Message.uncached { status_of(Message.find(id)) } + # @rbs (?authorization_context: untyped) -> String + def status(authorization_context: nil) + status_of(SolidObjects.client.read_message(self, authorization_context:)) end - # @rbs () -> untyped - def result - Message.uncached { Message.find(id).result } + # @rbs (?authorization_context: untyped) -> untyped + def result(authorization_context: nil) + SolidObjects.client.read_message(self, authorization_context:).result! end - # @rbs () -> Outcome - def outcome + # @rbs (?authorization_context: untyped) -> Outcome + def outcome(authorization_context: nil) Message.uncached do - message = Message.find(id) + message = SolidObjects.client.read_message(self, authorization_context:) Outcome.new( status: status_of(message), result: Serialization.readonly_copy(message.result), diff --git a/lib/solid_objects/synchronous_invocation.rb b/lib/solid_objects/synchronous_invocation.rb index 25b6b0e..c70c47d 100644 --- a/lib/solid_objects/synchronous_invocation.rb +++ b/lib/solid_objects/synchronous_invocation.rb @@ -78,27 +78,7 @@ def load_message_at_deadline(message_reference) # @rbs (Message) -> untyped def completed_result(message) - raise_rejection(message) if message.rejected? - if message.dead? - raise MessageFailed.new( - "actor message failed permanently", - message_id: message.id, - details: message.error || {} - ) - end - - Serialization.readonly_copy(message.result) - end - - # @rbs (Message) -> bot - def raise_rejection(message) - rejection = message.rejection - raise Rejected.new( - code: rejection.fetch("code"), - message: rejection.fetch("message"), - details: rejection.fetch("details"), - message_id: message.id - ) + message.result! end # @rbs () -> ProcessRegistry diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb index 310c527..8345282 100644 --- a/lib/solid_objects/telemetry.rb +++ b/lib/solid_objects/telemetry.rb @@ -8,6 +8,7 @@ module Telemetry depth count errorName outcome retryable status effectId effectName reminderId occurrence outboxKind truncated broadcastId code role reason processId processKind ownerId componentCount byteCount thresholdBytes previousRunAt nextRunAt name commitAction payload + previousIntervalMilliseconds currentIntervalMilliseconds ].freeze ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds" }.freeze @@ -78,6 +79,11 @@ def event(name, payload) # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] def safe_attributes(payload) payload.each_with_object({}) do |(key, value), attributes| + value = value.to_s if key == :reason && value.is_a?(Symbol) + if %i[previous_interval current_interval].include?(key) && value.is_a?(Numeric) + attributes["#{key.to_s.camelize(:lower)}Milliseconds"] = value * 1000 + next + end name = key.to_s.camelize(:lower) name = ALIASES.fetch(name, name) next unless FIELDS.include?(name) diff --git a/sig/generated/lib/solid_objects/client.rbs b/sig/generated/lib/solid_objects/client.rbs index f378eb0..46c1ada 100644 --- a/sig/generated/lib/solid_objects/client.rbs +++ b/sig/generated/lib/solid_objects/client.rbs @@ -16,6 +16,9 @@ module SolidObjects # @rbs (MessageReference, timeout: Numeric, ?authorization_context: untyped) -> untyped def wait: (MessageReference, timeout: Numeric, ?authorization_context: untyped) -> untyped + # @rbs (MessageReference, ?authorization_context: untyped) -> Message + def read_message: (MessageReference, ?authorization_context: untyped) -> Message + # @rbs (?reference: Reference?, ?request_id: String?, ?idempotency_key: String?, ?authorization_context: untyped) -> MessageReference? def find_by: (?reference: Reference?, ?request_id: String?, ?idempotency_key: String?, ?authorization_context: untyped) -> MessageReference? diff --git a/sig/generated/lib/solid_objects/message_reference.rbs b/sig/generated/lib/solid_objects/message_reference.rbs index 44390a0..097bde6 100644 --- a/sig/generated/lib/solid_objects/message_reference.rbs +++ b/sig/generated/lib/solid_objects/message_reference.rbs @@ -28,14 +28,14 @@ module SolidObjects # @rbs (id: Integer, request_id: String, actor_type: String, actor_id: String, sequence: Integer) -> void def initialize: (id: Integer, request_id: String, actor_type: String, actor_id: String, sequence: Integer) -> void - # @rbs () -> String - def status: () -> String + # @rbs (?authorization_context: untyped) -> String + def status: (?authorization_context: untyped) -> String - # @rbs () -> untyped - def result: () -> untyped + # @rbs (?authorization_context: untyped) -> untyped + def result: (?authorization_context: untyped) -> untyped - # @rbs () -> Outcome - def outcome: () -> Outcome + # @rbs (?authorization_context: untyped) -> Outcome + def outcome: (?authorization_context: untyped) -> Outcome # @rbs (?timeout: Numeric, ?authorization_context: untyped) -> untyped def wait: (?timeout: Numeric, ?authorization_context: untyped) -> untyped diff --git a/sig/generated/lib/solid_objects/synchronous_invocation.rbs b/sig/generated/lib/solid_objects/synchronous_invocation.rbs index ab11534..4a06d20 100644 --- a/sig/generated/lib/solid_objects/synchronous_invocation.rbs +++ b/sig/generated/lib/solid_objects/synchronous_invocation.rbs @@ -24,9 +24,6 @@ module SolidObjects # @rbs (Message) -> untyped def completed_result: (Message) -> untyped - # @rbs (Message) -> bot - def raise_rejection: (Message) -> bot - # @rbs () -> ProcessRegistry def process_registry: () -> ProcessRegistry diff --git a/sig/generated/models/solid_objects/message.rbs b/sig/generated/models/solid_objects/message.rbs index 0d6875e..9d9e572 100644 --- a/sig/generated/models/solid_objects/message.rbs +++ b/sig/generated/models/solid_objects/message.rbs @@ -17,6 +17,9 @@ module SolidObjects # @rbs () -> bool def dead?: () -> bool + # @rbs () -> untyped + def result!: () -> untyped + private # @rbs () -> void diff --git a/test/integration/result_lookup_test.rb b/test/integration/result_lookup_test.rb index 2477e47..7bf608f 100644 --- a/test/integration/result_lookup_test.rb +++ b/test/integration/result_lookup_test.rb @@ -19,6 +19,11 @@ def checkout(order_id:) { "order_id" => order_id } end + def oversized_result + self.items += 1 + "x" * 128 + end + def reject_checkout reject("closed", "the cart is closed") end @@ -34,6 +39,68 @@ def reject_checkout CartActor.fail = false end + test "reauthorizes every message read with the original operation and arguments" do + reference = CartActor.ref("alice") + reference.sync(idempotency_key: "protected").checkout(order_id: 42) + message = reference.find_by(idempotency_key: "protected") + reads = [] + SolidObjects.configuration.authorize_message = ->(operation:, arguments:, authorization_context:, **) do + reads << [ operation, arguments, authorization_context ] + authorization_context == "operator" + end + + %i[status result outcome].each do |method| + assert_raises(SolidObjects::Unauthorized) { message.public_send(method) } + assert message.public_send(method, authorization_context: "operator") + end + assert_equal 6, reads.length + assert reads.all? { |operation, arguments, _| operation == "checkout" && arguments == { "order_id" => 42 } } + SolidObjects.configuration.authorize_message = ->(**) { false } + %i[status result outcome].each do |method| + assert_raises(SolidObjects::Unauthorized) { message.public_send(method, authorization_context: "operator") } + end + end + + test "query result reads use the query policy" do + CartActor.ref("alice").sync(idempotency_key: "query").total + message = CartActor.ref("alice").find_by(idempotency_key: "query") + SolidObjects.configuration.authorize_query = ->(**) { false } + %i[status result outcome].each do |method| + assert_raises(SolidObjects::Unauthorized) { message.public_send(method) } + end + end + + test "message reads refuse forged invocation identity" do + CartActor.ref("alice").sync.checkout(order_id: 42) + stored = SolidObjects::Message.sole + forged = SolidObjects::MessageReference.new(id: stored.id, request_id: "wrong", actor_type: stored.actor_type, actor_id: stored.actor_id, sequence: stored.sequence) + %i[status result outcome].each do |method| + assert_raises(SolidObjects::Unauthorized) { forged.public_send(method) } + end + end + + test "result raises terminal failures while outcome remains inspectable" do + CartActor.fail = true + message = CartActor.ref("alice").async.checkout(order_id: 1) + run_actors + assert_raises(SolidObjects::MessageFailed) { message.result } + assert_equal "dead", message.outcome.status + rejected = CartActor.ref("alice").async.reject_checkout + run_actors + assert_raises(SolidObjects::Rejected) { rejected.result } + assert_equal "rejected", rejected.outcome.status + end + + test "background result size limits roll back the actor state" do + SolidObjects.configuration.max_result_bytes = 32 + reference = CartActor.ref("size-limit") + message = reference.async.oversized_result + run_actors + assert_raises(SolidObjects::MessageFailed) { message.result } + assert_equal "SolidObjects::PayloadTooLarge", message.outcome.error.class_name + assert_equal 0, reference.snapshot.items + end + test "finds a completed message by request id and reads its result" do CartActor.ref("alice").sync.checkout(order_id: 4210) original = SolidObjects::Message.sole @@ -48,14 +115,15 @@ def reject_checkout assert_equal({ "order_id" => 4210 }, found.result) end - test "reports no result for a message that was enqueued asynchronously" do + test "retains the result of a message that was enqueued asynchronously" do original = CartActor.ref("alice").async.checkout(order_id: 4210) run_actors found = SolidObjects.client.find_by(request_id: original.request_id) assert_equal "completed", found.status - assert_nil found.result + assert_equal({ "order_id" => 4210 }, found.result) + assert found.result.frozen? end test "finds a completed message by idempotency key on its reference" do diff --git a/test/integration/retry_test.rb b/test/integration/retry_test.rb index ba954b1..5b5152e 100644 --- a/test/integration/retry_test.rb +++ b/test/integration/retry_test.rb @@ -44,7 +44,7 @@ def continue_processing message = SolidObjects::Message.find(message_reference.id) assert_equal 2, message.attempt_count assert message.completed? - assert_nil message.result + assert_equal 1, message.result assert_equal({ "executions" => 1 }, message.instance.state) ensure worker&.stop diff --git a/test/integration/synchronous_invocation_test.rb b/test/integration/synchronous_invocation_test.rb index 875c599..cb7558b 100644 --- a/test/integration/synchronous_invocation_test.rb +++ b/test/integration/synchronous_invocation_test.rb @@ -391,7 +391,8 @@ def wait(timeout:) assert_raises(SolidObjects::Unauthorized) do message_reference.wait(timeout: 1) end - assert_equal "ready", message_reference.status + assert_raises(SolidObjects::Unauthorized) { message_reference.status } + assert SolidObjects::Message.find(message_reference.id).ready? end test "sync database lock waits are bounded by the invocation deadline" do diff --git a/test/integration/telemetry_test.rb b/test/integration/telemetry_test.rb index 8e0e537..498aa73 100644 --- a/test/integration/telemetry_test.rb +++ b/test/integration/telemetry_test.rb @@ -22,6 +22,14 @@ def increment end end + test "portable polling transitions keep millisecond intervals and reasons" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + backoff = SolidObjects::PollingBackoff.new(minimum_interval: 0.1, maximum_interval: 1, on_change: ->(transition) { SolidObjects.instrument(:"polling.interval_changed", role: "actors", **transition) }) + backoff.record_idle + assert_equal({ "role" => "actors", "previousIntervalMilliseconds" => 100, "currentIntervalMilliseconds" => 200, "reason" => "idle" }, events.last.fetch("attributes")) + end + test "a failing started subscriber cannot fail a turn" do subscriber = ActiveSupport::Notifications.subscribe("solid_objects.message.started") { raise "private sink failure" } assert_equal 1, Counter.ref("one").increment diff --git a/test/integration/transmit_staging_test.rb b/test/integration/transmit_staging_test.rb index c5cdf66..ceb71c7 100644 --- a/test/integration/transmit_staging_test.rb +++ b/test/integration/transmit_staging_test.rb @@ -21,6 +21,11 @@ def increment_mirror(amount:) actorId: "mirror-1" end + def stage_pair + transmit.increment(amount: 1) + transmit.increment(amount: 2) + end + def stage_invalid emit "solid-objects.transmit", arguments: { amount: 1 } end @@ -158,6 +163,22 @@ def deliver_to_mirror(envelope) assert_equal({ "count" => 3, "applied" => [ 1, 2 ] }, mirror_state) end + test "keeps staging order within one turn across a failed delivery" do + failures_remaining = 1 + SolidObjects.register_transmit do |envelope| + if envelope.dig("arguments", "amount") == 1 && failures_remaining.positive? + failures_remaining -= 1 + raise "network down" + end + deliver_to_mirror(envelope) + end + CounterActor.ref("alice").async.stage_pair + worker.run_until_idle + drain_effects + worker.run_until_idle + assert_equal [ 1, 2 ], mirror_state.fetch("applied") + end + test "a later claimed effect delivers an undelivered earlier sibling first" do delivered = [] SolidObjects.register_transmit do |envelope| From 4530819cf55dbe57c1c7fcc5b87e33f77a588be4 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 11:22:30 -0700 Subject: [PATCH 04/11] fix: Keep queries and projections read-only Reject actor state changes and durable intents from queries and observable projections before they can commit. Fail these violations without retrying, matching the JavaScript runtime. Pin reserved JSON names with shared fixtures and correct reminder limits and dead-transmit recovery documentation. --- CHANGELOG.md | 6 ++ compatibility/json-values.json | 14 +++ docs/architecture.md | 10 +- docs/reminders.md | 4 +- docs/roadmap.md | 3 + docs/transmission.md | 18 +++- lib/solid_objects/actor.rb | 24 ++++- lib/solid_objects/errors.rb | 3 + lib/solid_objects/executor.rb | 9 +- sig/generated/lib/solid_objects/actor.rbs | 6 ++ sig/generated/lib/solid_objects/errors.rbs | 3 + sig/generated/lib/solid_objects/executor.rbs | 2 +- test/integration/read_only_actor_test.rb | 97 ++++++++++++++++++++ test/integration/state_commit_test.rb | 4 +- test/unit/serialization_test.rb | 9 ++ 15 files changed, 192 insertions(+), 20 deletions(-) create mode 100644 compatibility/json-values.json create mode 100644 test/integration/read_only_actor_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index bd87df4..97d06d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## Unreleased +- Reject query and observable state mutation and staged durable work with + terminal `QueryMutatedState` errors. Cover individual snapshot projections and + preserve ordinary operations' already-staged work while reading projections. +- Pin reserved JSON property names with shared Ruby/JS fixtures. Document the + reminder-name limit difference and the authorized dead-transmit retry API. + - Reauthorize every message-reference status, result, and outcome read against the original invocation. Pass `authorization_context:` on every read. - Retain immutable JSON results for background and internal messages as well as diff --git a/compatibility/json-values.json b/compatibility/json-values.json new file mode 100644 index 0000000..9e2b191 --- /dev/null +++ b/compatibility/json-values.json @@ -0,0 +1,14 @@ +[ + { + "name": "object prototype key", + "value": { "__proto__": { "role": "ordinary data" }, "value": 1 } + }, + { "name": "scalar prototype key", "value": { "__proto__": "ordinary data" } }, + { "name": "null prototype key", "value": { "__proto__": null } }, + { + "name": "nested reserved names", + "value": { + "items": [{ "__proto__": { "constructor": "data" }, "prototype": true, "hasOwnProperty": 1 }] + } + } +] diff --git a/docs/architecture.md b/docs/architecture.md index 5a40d02..ea8540b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -167,10 +167,12 @@ creating a message or activation, applies required state migrations in memory, and returns deeply frozen declared attributes. It can race with an in-flight turn. `SolidObjects.mutable_copy` creates an independent mutable JSON value. -`message` and `query` both execute as durable mailbox turns. A query may not -mutate state. The executor detects query mutation and fails the message. An -observable is a named projection of state used by server rendering and realtime -updates. Its durable broadcast row stores only an empty invalidation marker by +`message` and `query` both execute as durable mailbox turns. Queries and +observables must not mutate state or stage effects, recovery checks, commit +actions, reminders, or outbound messages. Violations raise `QueryMutatedState` +and fail the message without retrying or committing its work. Individual snapshot +projections enforce the same rule. An observable is a named projection used by +server rendering and realtime updates. Its durable broadcast row stores only an empty invalidation marker by default. `broadcast: :value` explicitly opts into storing and sharing the projected value. diff --git a/docs/reminders.md b/docs/reminders.md index 2f51d1a..32c56d7 100644 --- a/docs/reminders.md +++ b/docs/reminders.md @@ -65,7 +65,9 @@ key only decides which alarm is which. A key must be non-empty, and the name it becomes must fit the 191-character column, which is checked on the composed name rather than the key alone so a -long operation and a short key are caught too. +long operation and a short key are caught too. JavaScript allows 255 UTF-16 code +units for the same combined name. These existing schema limits remain different; +use at most 191 ASCII characters for names shared across runtimes. The key is separated from the operation by a colon, so an operation may not hold one. Otherwise an unkeyed `deliver:item` and a `deliver` keyed `item` would be diff --git a/docs/roadmap.md b/docs/roadmap.md index b77682c..f86be8e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -7,6 +7,9 @@ - Rails engine, install generator, migration, and CLI - Explicit actor registry, references, JSON state, and state migrations +- Queries and observable projections reject state mutation and staged durable + work with terminal `QueryMutatedState` errors, matching JavaScript. Shared JSON + fixtures preserve reserved property names as ordinary data in both runtimes. - Fluent direct synchronous RPC, configured `sync`, and durable `async` - Durable message history plus ready/claimed membership tables - Concurrent sequence allocation and actor creation. An enqueue finds the diff --git a/docs/transmission.md b/docs/transmission.md index b199f3e..ed8d4dc 100644 --- a/docs/transmission.md +++ b/docs/transmission.md @@ -87,11 +87,19 @@ SolidObjects.configure do |configuration| end ``` -These settings apply to every effect, not only transmits. A dead transmit -effect has no retry API; the dashboard lists it, and recovery means -returning its row to `pending` with a cleared `attempt_count`. Order -survives that recovery, because the drain orders by mailbox sequence, not -by retry time. +These settings apply to every effect, including transmits. Retry a dead transmit +through the authorized administration API: + +```ruby +SolidObjects.dead_letters.effects.retry(effect_id, authorization_context: operator) +``` + +Retry resets attempts and returns the effect to pending with its stable identity, +so the receiver still deduplicates replays. The administration policy must allow +`retry` on `effect_dead_letters`; the action is recorded in the audit log. Use +`SolidObjects.dead_letters.effects.redrive(authorization_context: operator)` to +recover a scope in bounded batches. Order survives recovery because the drain +orders by source sequence and staging order. ## Wire contract diff --git a/lib/solid_objects/actor.rb b/lib/solid_objects/actor.rb index 455d28b..94dcaeb 100644 --- a/lib/solid_objects/actor.rb +++ b/lib/solid_objects/actor.rb @@ -459,7 +459,9 @@ def invoke(operation, arguments) # @rbs () -> Hash[String, untyped] def observable_values - guard_application_writes("observables") do + return {} if self.class.definition.observables.empty? + + read_projection("observables") do self.class.definition.observables.each_with_object({}) do |(name, handler), values| values[name.to_s] = Serialization.dump(instance_exec(&handler.block)) end @@ -472,7 +474,7 @@ def observable_value(name) handler = self.class.definition.observables[observable_name] raise UnknownMessage, "unknown observable #{name.inspect}" unless handler - guard_application_writes("observable.#{observable_name}") do + read_projection("observable.#{observable_name}") do Serialization.dump(instance_exec(&handler.block)) end end @@ -537,6 +539,12 @@ def discard_intents outbound_message_intents.clear end + # @rbs () -> Integer + def intent_count + effect_intents.length + effect_recovery_intents.length + commit_action_intents.length + + reminder_intents.length + outbound_message_intents.length + end + private attr_reader :effect_intents, @@ -545,6 +553,18 @@ def discard_intents :reminder_intents, :outbound_message_intents + # @rbs (String) { () -> untyped } -> untyped + def read_projection(operation) + state_before = state.to_h + intents_before = intent_count + result = guard_application_writes(operation) { yield } + unless state.to_h == state_before && intent_count == intents_before + raise QueryMutatedState, "observables must not mutate actor state or stage durable work" + end + + result + end + # @rbs (String) { () -> untyped } -> untyped def guard_application_writes(operation, &block) ApplicationWriteGuard.call( diff --git a/lib/solid_objects/errors.rb b/lib/solid_objects/errors.rb index dbad8da..ed05099 100644 --- a/lib/solid_objects/errors.rb +++ b/lib/solid_objects/errors.rb @@ -7,6 +7,9 @@ class Error < StandardError class NonRetryableError < Error end + class QueryMutatedState < NonRetryableError + end + class UnsupportedDatabase < Error end diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index d5cf11b..060937e 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -31,7 +31,7 @@ def call result = invoke_actor(message_context) observable_changes = changed_observables(observables_before, actor.observable_values) state_after = actor.state.to_h_with_byte_size - ensure_query_did_not_mutate_state!(state_before, state_after.value) + ensure_query_is_read_only!(state_before, state_after.value) complete( result, observable_changes, @@ -70,12 +70,11 @@ def invoke_actor(message_context) end # @rbs (Hash[String, untyped], Hash[String, untyped]) -> void - def ensure_query_did_not_mutate_state!(state_before, state_after) - return unless message.delivery_mode == "sync" + def ensure_query_is_read_only!(state_before, state_after) return unless actor.class.definition.queries.key?(message.operation.to_sym) - return if state_after == state_before - raise InvalidActor, "query #{message.operation.inspect} mutated actor state" + raise QueryMutatedState, "query #{message.operation.inspect} mutated actor state" if state_after != state_before + raise QueryMutatedState, "query #{message.operation.inspect} staged durable work" if actor.intent_count.positive? end # @rbs (Hash[String, untyped], Hash[String, untyped]) -> Hash[String, untyped] diff --git a/sig/generated/lib/solid_objects/actor.rbs b/sig/generated/lib/solid_objects/actor.rbs index 5447898..bb38507 100644 --- a/sig/generated/lib/solid_objects/actor.rbs +++ b/sig/generated/lib/solid_objects/actor.rbs @@ -348,6 +348,9 @@ module SolidObjects # @rbs () -> void def discard_intents: () -> void + # @rbs () -> Integer + def intent_count: () -> Integer + private attr_reader effect_intents: untyped @@ -360,6 +363,9 @@ module SolidObjects attr_reader outbound_message_intents: untyped + # @rbs (String) { () -> untyped } -> untyped + def read_projection: (String) { () -> untyped } -> untyped + # @rbs (String) { () -> untyped } -> untyped def guard_application_writes: (String) { () -> untyped } -> untyped diff --git a/sig/generated/lib/solid_objects/errors.rbs b/sig/generated/lib/solid_objects/errors.rbs index 53c74be..93ea002 100644 --- a/sig/generated/lib/solid_objects/errors.rbs +++ b/sig/generated/lib/solid_objects/errors.rbs @@ -7,6 +7,9 @@ module SolidObjects class NonRetryableError < Error end + class QueryMutatedState < NonRetryableError + end + class UnsupportedDatabase < Error end diff --git a/sig/generated/lib/solid_objects/executor.rbs b/sig/generated/lib/solid_objects/executor.rbs index 95f7bd9..22d68ae 100644 --- a/sig/generated/lib/solid_objects/executor.rbs +++ b/sig/generated/lib/solid_objects/executor.rbs @@ -29,7 +29,7 @@ module SolidObjects def invoke_actor: (MessageContext) -> untyped # @rbs (Hash[String, untyped], Hash[String, untyped]) -> void - def ensure_query_did_not_mutate_state!: (Hash[String, untyped], Hash[String, untyped]) -> void + def ensure_query_is_read_only!: (Hash[String, untyped], Hash[String, untyped]) -> void # @rbs (Hash[String, untyped], Hash[String, untyped]) -> Hash[String, untyped] def changed_observables: (Hash[String, untyped], Hash[String, untyped]) -> Hash[String, untyped] diff --git a/test/integration/read_only_actor_test.rb b/test/integration/read_only_actor_test.rb new file mode 100644 index 0000000..2086e90 --- /dev/null +++ b/test/integration/read_only_actor_test.rb @@ -0,0 +1,97 @@ +# frozen_string_literal: true + +require "database_test_helper" + +class ReadOnlyActorTest < ActiveSupport::TestCase + class Reader < SolidObjects::Actor + actor_type "read-only-reader" + attribute :items, default: [] + + class << self + attr_accessor :projection_action + end + + query :read do |action:| + perform_action(action) + items + end + + observable :projected do + perform_action(self.class.projection_action) if self.class.projection_action + items + end + + def append + items << "committed" + items + end + + private + + def perform_action(action) + case action + when "effect" then emit :unexpected + when "recovery" then request_effect_recovery("effect_id" => "missing-effect") + when "commit_action" then commit_action :unexpected + when "reminder" then schedule(at: Time.now + 60).append + when "outbound" then send_to(self.class.ref("other")).append + when "state" then items << "unexpected" + end + end + end + + setup do + Reader.projection_action = nil + SolidObjects.configuration.max_attempts = 3 + SolidObjects.configuration.retry_delay = ->(_) { 0 } + SolidObjects.register_commit_action(:unexpected) { SolidObjectsTestDomainRecord.create!(name: "unexpected") } + end + + %w[effect recovery commit_action reminder outbound state].each do |action| + test "queries reject #{action} without committing or retrying" do + error = assert_raises(SolidObjects::MessageFailed) { Reader.ref("one").sync.read(action:) } + + assert_equal "SolidObjects::QueryMutatedState", error.details.fetch("class") + assert_no_committed_work + end + + test "observables reject #{action} without committing or retrying" do + Reader.projection_action = action + + error = assert_raises(SolidObjects::MessageFailed) { Reader.ref("one").sync.append } + + assert_equal "SolidObjects::QueryMutatedState", error.details.fetch("class") + assert_no_committed_work + end + end + + test "individual snapshot projections cannot mutate state" do + Reader.projection_action = "state" + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("snapshot")) + + error = assert_raises(SolidObjects::Error) { snapshot.observable_value(:projected) } + + assert_equal "SolidObjects::QueryMutatedState", error.class.name + assert_empty SolidObjects::Instance.all + end + + test "pure projections preserve effects already staged by an operation" do + reader = Reader.new(actor_id: "local", state: SolidObjects::State.new(Reader.definition.state_definition)) + reader.emit(:expected) + + assert_equal({ "projected" => [] }, reader.observable_values) + assert_equal "expected", reader.drain_effect_intents.sole.name + end + + private + + def assert_no_committed_work + assert_empty SolidObjects::Instance.find_by!(actor_id: "one").state + assert_equal 1, SolidObjects::Message.sole.attempt_count + assert_empty SolidObjects::Effect.all + assert_empty SolidObjects::EffectRecovery.all + assert_empty SolidObjects::Reminder.all + assert_empty SolidObjects::Broadcast.all + assert_empty SolidObjectsTestDomainRecord.all + end +end diff --git a/test/integration/state_commit_test.rb b/test/integration/state_commit_test.rb index 20c751c..8c7457d 100644 --- a/test/integration/state_commit_test.rb +++ b/test/integration/state_commit_test.rb @@ -151,7 +151,7 @@ def respond_to_missing?(*) CountingActor.ref("guard").sync.mutating end - assert_equal "SolidObjects::InvalidActor", error.details.fetch("class") + assert_equal "SolidObjects::QueryMutatedState", error.details.fetch("class") instance = SolidObjects::Instance.find_by!(actor_type: "state-commit-counting", actor_id: "guard") assert_empty instance.state, "the rejected mutation must not reach the committed row" end @@ -163,7 +163,7 @@ def respond_to_missing?(*) ObservableMutatingActor.ref("observable").sync.current end - assert_equal "SolidObjects::InvalidActor", error.details.fetch("class") + assert_equal "SolidObjects::QueryMutatedState", error.details.fetch("class") instance = SolidObjects::Instance.find_by!(actor_type: "state-commit-observable", actor_id: "observable") assert_empty instance.state, "the observable mutation must not reach the committed row" end diff --git a/test/unit/serialization_test.rb b/test/unit/serialization_test.rb index 625d085..3594523 100644 --- a/test/unit/serialization_test.rb +++ b/test/unit/serialization_test.rb @@ -3,6 +3,15 @@ require "test_helper" class SerializationTest < ActiveSupport::TestCase + JSON.parse(File.read(File.expand_path("../../compatibility/json-values.json", __dir__))).each do |fixture| + test "preserves #{fixture.fetch("name")} as JSON data" do + value = fixture.fetch("value") + + assert_equal value, SolidObjects::Serialization.dump(value) + assert_equal value, SolidObjects::Serialization.readonly_copy(value) + end + end + test "normalizes symbol keys and nested values" do value = SolidObjects::Serialization.dump({ product_id: "shirt", From d9667bd0a2d34230efbcbd8ceedfd5da693d47d7 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 11:34:59 -0700 Subject: [PATCH 05/11] fix: Detect replaced work in projections Compare complete staged work around projections so draining one intent and staging another cannot bypass the read-only contract. Cover effect and commit-action replacement with terminal rollback regressions. --- CHANGELOG.md | 3 ++- lib/solid_objects/actor.rb | 11 +++++++++-- sig/generated/lib/solid_objects/actor.rbs | 3 +++ test/integration/read_only_actor_test.rb | 23 +++++++++++++++++++++++ 4 files changed, 37 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 97d06d6..02c9a3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,8 @@ - Reject query and observable state mutation and staged durable work with terminal `QueryMutatedState` errors. Cover individual snapshot projections and - preserve ordinary operations' already-staged work while reading projections. + preserve ordinary operations' already-staged work while reading projections, + including replacements that leave the intent count unchanged. - Pin reserved JSON property names with shared Ruby/JS fixtures. Document the reminder-name limit difference and the authorized dead-transmit retry API. diff --git a/lib/solid_objects/actor.rb b/lib/solid_objects/actor.rb index 94dcaeb..6afd5ae 100644 --- a/lib/solid_objects/actor.rb +++ b/lib/solid_objects/actor.rb @@ -556,15 +556,22 @@ def intent_count # @rbs (String) { () -> untyped } -> untyped def read_projection(operation) state_before = state.to_h - intents_before = intent_count + intents_before = intent_snapshot result = guard_application_writes(operation) { yield } - unless state.to_h == state_before && intent_count == intents_before + unless state.to_h == state_before && intent_snapshot == intents_before raise QueryMutatedState, "observables must not mutate actor state or stage durable work" end result end + # @rbs () -> Array[Array[Hash[Symbol, untyped]]] + def intent_snapshot + [ effect_intents, effect_recovery_intents, commit_action_intents, reminder_intents, outbound_message_intents ].map do |intents| + intents.map { |intent| intent.to_h.deep_dup } + end + end + # @rbs (String) { () -> untyped } -> untyped def guard_application_writes(operation, &block) ApplicationWriteGuard.call( diff --git a/sig/generated/lib/solid_objects/actor.rbs b/sig/generated/lib/solid_objects/actor.rbs index bb38507..0f6648f 100644 --- a/sig/generated/lib/solid_objects/actor.rbs +++ b/sig/generated/lib/solid_objects/actor.rbs @@ -366,6 +366,9 @@ module SolidObjects # @rbs (String) { () -> untyped } -> untyped def read_projection: (String) { () -> untyped } -> untyped + # @rbs () -> Array[Array[Hash[Symbol, untyped]]] + def intent_snapshot: () -> Array[Array[Hash[Symbol, untyped]]] + # @rbs (String) { () -> untyped } -> untyped def guard_application_writes: (String) { () -> untyped } -> untyped diff --git a/test/integration/read_only_actor_test.rb b/test/integration/read_only_actor_test.rb index 2086e90..96973c4 100644 --- a/test/integration/read_only_actor_test.rb +++ b/test/integration/read_only_actor_test.rb @@ -26,6 +26,11 @@ def append items end + def append_with_work(action:) + perform_action(action) + append + end + private def perform_action(action) @@ -36,6 +41,12 @@ def perform_action(action) when "reminder" then schedule(at: Time.now + 60).append when "outbound" then send_to(self.class.ref("other")).append when "state" then items << "unexpected" + when "replace_effect", "replace_commit_action" + return unless intent_count.positive? + + discard_intents + emit(:replacement) if action == "replace_effect" + commit_action(:replacement) if action == "replace_commit_action" end end end @@ -45,6 +56,7 @@ def perform_action(action) SolidObjects.configuration.max_attempts = 3 SolidObjects.configuration.retry_delay = ->(_) { 0 } SolidObjects.register_commit_action(:unexpected) { SolidObjectsTestDomainRecord.create!(name: "unexpected") } + SolidObjects.register_commit_action(:replacement) { SolidObjectsTestDomainRecord.create!(name: "replacement") } end %w[effect recovery commit_action reminder outbound state].each do |action| @@ -75,6 +87,17 @@ def perform_action(action) assert_empty SolidObjects::Instance.all end + %w[effect commit_action].each do |action| + test "observables cannot replace staged #{action} with the same intent count" do + Reader.projection_action = "replace_#{action}" + + error = assert_raises(SolidObjects::MessageFailed) { Reader.ref("one").sync.append_with_work(action:) } + + assert_equal "SolidObjects::QueryMutatedState", error.details.fetch("class") + assert_no_committed_work + end + end + test "pure projections preserve effects already staged by an operation" do reader = Reader.new(actor_id: "local", state: SolidObjects::State.new(Reader.definition.state_definition)) reader.emit(:expected) From 6771a7607e1e8e9fd4fcb9fa17a2a8c096dac9da Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Wed, 30 Sep 2026 13:12:35 -0700 Subject: [PATCH 06/11] fix: Align payload and timeout behavior Guard and isolate payload projections, honor configured byte limits, and preserve generated defaults within each committed snapshot. Normalize timeout telemetry to the shared JavaScript contract. Let SQLite lock holders run during background busy waits on older Rails versions, reinstalling the yielding handler on each attempt. Cover the behavior with shared fixtures, real adapter regressions, and reversal checks. --- CHANGELOG.md | 9 ++ compatibility/sync-timeout.json | 10 ++ docs/observability.md | 11 ++ docs/roadmap.md | 15 ++- lib/solid_objects/actor.rb | 18 +-- lib/solid_objects/actor_snapshot.rb | 22 ++-- lib/solid_objects/database_adapters/sqlite.rb | 8 +- lib/solid_objects/payload_broadcast.rb | 17 +-- lib/solid_objects/telemetry.rb | 4 +- sig/generated/lib/solid_objects/actor.rbs | 6 +- .../lib/solid_objects/actor_snapshot.rbs | 9 +- .../lib/solid_objects/payload_broadcast.rbs | 6 +- test/integration/payload_projection_test.rb | 103 ++++++++++++++++++ test/integration/sqlite_busy_wait_test.rb | 53 +++++++++ .../synchronous_invocation_test.rb | 6 + test/integration/telemetry_test.rb | 26 +++++ 16 files changed, 282 insertions(+), 41 deletions(-) create mode 100644 compatibility/sync-timeout.json create mode 100644 test/integration/payload_projection_test.rb create mode 100644 test/integration/sqlite_busy_wait_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 02c9a3c..d55f3c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ ## Unreleased +- Guard personalized payload projections against state changes, staged work, + and application database writes. Each payload gets an isolated actor from + the committed snapshot and honors `max_payload_bytes`, matching JavaScript. +- Preserve timeout wait reasons, activation owner IDs, and activation generations + in portable telemetry using the shared camelCase fields and reason values. +- Use a yielding SQLite busy handler for background transactions so concurrent + writers can finish on Rails 7.1 and 7.2. Preserve configured wait limits and + synchronous deadlines; cover contention with a coordinated lock regression. + - Reject query and observable state mutation and staged durable work with terminal `QueryMutatedState` errors. Cover individual snapshot projections and preserve ordinary operations' already-staged work while reading projections, diff --git a/compatibility/sync-timeout.json b/compatibility/sync-timeout.json new file mode 100644 index 0000000..a851215 --- /dev/null +++ b/compatibility/sync-timeout.json @@ -0,0 +1,10 @@ +[ + { "rubyReason": "actor_paused", "waitingOn": "actorPaused" }, + { "rubyReason": "activation_held", "waitingOn": "activationHeld" }, + { "rubyReason": "earlier_message", "waitingOn": "earlierMessage" }, + { "rubyReason": "message_claimed", "waitingOn": "messageClaimed" }, + { "rubyReason": "not_yet_available", "waitingOn": "notYetAvailable" }, + { "rubyReason": "ready_unclaimed", "waitingOn": "readyUnclaimed" }, + { "rubyReason": "database_contention", "waitingOn": "databaseContention" }, + { "rubyReason": "unknown", "waitingOn": "unknown" } +] diff --git a/docs/observability.md b/docs/observability.md index ff19d2d..4e893b6 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -47,6 +47,17 @@ Portable `polling.interval_changed` events carry `previousIntervalMilliseconds`, `currentIntervalMilliseconds`, and a string `reason`. Ruby converts its native second-based notification values while preserving the original notification. +## Synchronous timeout diagnostics + +`solid_objects.sync.timeout` includes `waitingOn`, `activationOwnerId`, and +`activationGeneration` in `attributes`. Generations are decimal strings; +unavailable activation fields are null. Both runtimes use the same wait reasons: +`actorPaused`, `activationHeld`, `earlierMessage`, `messageClaimed`, +`notYetAvailable`, `readyUnclaimed`, `databaseContention`, and `unknown`. +Ruby's exception attributes and Active Support notifications retain their native +snake_case names and reason values. The portable instrumentation envelope uses +the shared camelCase contract. + ## Metrics and tracing Metrics are sample descriptions. Exporting them is opt-in: the runtime does not diff --git a/docs/roadmap.md b/docs/roadmap.md index f86be8e..0e424f5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -3,13 +3,16 @@ ## Implemented and tested - Portable telemetry with a shared JSON schema, metric samples, isolated observer - callbacks, and bounded authorized actor diagnostics. See [observability](observability.md). + callbacks, bounded authorized actor diagnostics, and matching timeout wait + reasons and activation metadata. See [observability](observability.md). - Rails engine, install generator, migration, and CLI - Explicit actor registry, references, JSON state, and state migrations -- Queries and observable projections reject state mutation and staged durable - work with terminal `QueryMutatedState` errors, matching JavaScript. Shared JSON - fixtures preserve reserved property names as ordinary data in both runtimes. +- Queries, observable projections, and personalized payloads reject state mutation + and staged durable work with `QueryMutatedState`, matching JavaScript. Query + and observable violations fail terminally; a payload violation is confined to + that payload. Shared JSON fixtures preserve reserved property names as ordinary + data in both runtimes. - Fluent direct synchronous RPC, configured `sync`, and durable `async` - Durable message history plus ready/claimed membership tables - Concurrent sequence allocation and actor creation. An enqueue finds the @@ -61,7 +64,9 @@ subscriber's authorization context, fenced by actor revision, resolved through `payload_authorization_context` so the block and `authorize_query` see the same subject a controller render passes, and confined so one failing payload - cannot reject the subscription or stop its siblings + cannot reject the subscription or stop its siblings. Each payload uses an + isolated actor from the same committed snapshot, prevents application database + writes, and enforces the configured `max_payload_bytes` limit - Reconciliation read APIs - Installation doctor, authorization reference, fit guide, and legacy-state migration cookbook. The doctor names every column that a migration after the diff --git a/lib/solid_objects/actor.rb b/lib/solid_objects/actor.rb index 6afd5ae..ffc5336 100644 --- a/lib/solid_objects/actor.rb +++ b/lib/solid_objects/actor.rb @@ -545,26 +545,26 @@ def intent_count reminder_intents.length + outbound_message_intents.length end - private - - attr_reader :effect_intents, - :effect_recovery_intents, - :commit_action_intents, - :reminder_intents, - :outbound_message_intents - # @rbs (String) { () -> untyped } -> untyped def read_projection(operation) state_before = state.to_h intents_before = intent_snapshot result = guard_application_writes(operation) { yield } unless state.to_h == state_before && intent_snapshot == intents_before - raise QueryMutatedState, "observables must not mutate actor state or stage durable work" + raise QueryMutatedState, "projections must not mutate actor state or stage durable work" end result end + private + + attr_reader :effect_intents, + :effect_recovery_intents, + :commit_action_intents, + :reminder_intents, + :outbound_message_intents + # @rbs () -> Array[Array[Hash[Symbol, untyped]]] def intent_snapshot [ effect_intents, effect_recovery_intents, commit_action_intents, reminder_intents, outbound_message_intents ].map do |intents| diff --git a/lib/solid_objects/actor_snapshot.rb b/lib/solid_objects/actor_snapshot.rb index 2df7dc4..c2100a1 100644 --- a/lib/solid_objects/actor_snapshot.rb +++ b/lib/solid_objects/actor_snapshot.rb @@ -7,6 +7,7 @@ class ActorSnapshot # @rbs @actor: Actor # @rbs @instance_id: Integer # @rbs @revision: Integer + # @rbs @state_data: Hash[String, untyped] # @rbs @observable_values: Hash[String, untyped]? # @rbs @observable_value_cache: Hash[String, untyped] @@ -24,6 +25,7 @@ def initialize(reference) end @instance_id = @instance&.id || 0 @revision = @instance&.state_revision || 0 + @state_data = State.new(actor_class.definition.state_definition, migrated_state).to_h @actor = build_actor @observable_values = nil @observable_value_cache = {} @@ -43,14 +45,23 @@ def observable_value(name) Serialization.readonly_copy(actor.observable_value(observable_name)) end + # @rbs () -> Actor + def build_actor + actor_class.new( + actor_id: reference.actor_id, + state: State.new(actor_class.definition.state_definition, @state_data), + instance_id: instance&.id + ) + end + private attr_reader :instance - # @rbs () -> Actor - def build_actor + # @rbs () -> Hash[String, untyped] + def migrated_state state_version = instance&.state_version || actor_class.state_version - state_data = ApplicationWriteGuard.call( + ApplicationWriteGuard.call( actor_type: reference.actor_type, actor_id: reference.actor_id, operation: "state_migration" @@ -60,11 +71,6 @@ def build_actor instance&.state || {} ) end - actor_class.new( - actor_id: reference.actor_id, - state: State.new(actor_class.definition.state_definition, state_data), - instance_id: @instance&.id - ) end end end diff --git a/lib/solid_objects/database_adapters/sqlite.rb b/lib/solid_objects/database_adapters/sqlite.rb index ef1e3d5..dd6105d 100644 --- a/lib/solid_objects/database_adapters/sqlite.rb +++ b/lib/solid_objects/database_adapters/sqlite.rb @@ -22,7 +22,13 @@ def current_time_expression def transaction(&block) return with_lock_retry { super } if SyncDeadline.active? - with_busy_retry { super } + with_busy_retry do + with_connection do |connection| + timeout = configured_busy_handler_timeout(connection) + connection.raw_connection.busy_handler_timeout = timeout if timeout + super + end + end end # A write outside a synchronous deadline has no Ruby-level budget, so it diff --git a/lib/solid_objects/payload_broadcast.rb b/lib/solid_objects/payload_broadcast.rb index cf47ee8..d441d0b 100644 --- a/lib/solid_objects/payload_broadcast.rb +++ b/lib/solid_objects/payload_broadcast.rb @@ -2,7 +2,6 @@ module SolidObjects class PayloadBroadcast - MAXIMUM_PAYLOAD_BYTES = 1_048_576 REVISION_OBSERVABLE = "solid_objects.revision" # @rbs @snapshot: ActorSnapshot @@ -42,7 +41,7 @@ def call def rendered_payload(handler) payload = Serialization.dump( evaluated_payload(handler), - max_bytes: MAXIMUM_PAYLOAD_BYTES + max_bytes: SolidObjects.configuration.max_payload_bytes ) return payload if payload.is_a?(Hash) || payload.is_a?(Array) @@ -56,10 +55,12 @@ def rendered_payload(handler) # unaffected. # @rbs (ActorDefinition::Handler) -> untyped def evaluated_payload(handler) - actor = snapshot.actor - actor.instance_exec(actor, authorization_context, &handler.block) + actor = snapshot.build_actor + actor.read_projection("payload.#{name}") do + actor.instance_exec(actor, authorization_context, &handler.block) + end rescue NameError => error - raise unless class_level_receiver?(error) + raise unless class_level_receiver?(error, actor) raise InvalidPayloadBroadcast, "payload broadcast #{name.inspect} called #{error.name.inspect} on the " \ @@ -70,9 +71,9 @@ def evaluated_payload(handler) # Distinguishes a block that relied on the old class-level receiver from an # ordinary typo, so the one behaviour change reports itself instead of # surfacing as an unexplained NameError. - # @rbs (NameError[untyped]) -> bool - def class_level_receiver?(error) - error.receiver.equal?(snapshot.actor) && + # @rbs (NameError[untyped], Actor?) -> bool + def class_level_receiver?(error, actor) + error.receiver.equal?(actor) && snapshot.actor_class.respond_to?(error.name) rescue ArgumentError, NameError false diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb index 8345282..daa305b 100644 --- a/lib/solid_objects/telemetry.rb +++ b/lib/solid_objects/telemetry.rb @@ -9,6 +9,7 @@ module Telemetry outboxKind truncated broadcastId code role reason processId processKind ownerId componentCount byteCount thresholdBytes previousRunAt nextRunAt name commitAction payload previousIntervalMilliseconds currentIntervalMilliseconds + waitingOn activationOwnerId activationGeneration ].freeze ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds" }.freeze @@ -89,7 +90,8 @@ def safe_attributes(payload) next unless FIELDS.include?(name) next unless value.nil? || value.is_a?(String) || value.is_a?(Numeric) || value == true || value == false - value = value.to_s if !value.nil? && (name.end_with?("Id") || %w[revision sequence generation].include?(name)) + value = value.camelize(:lower) if name == "waitingOn" && value.is_a?(String) + value = value.to_s if !value.nil? && (name.end_with?("Id") || %w[revision sequence generation activationGeneration].include?(name)) attributes[name] = value end end diff --git a/sig/generated/lib/solid_objects/actor.rbs b/sig/generated/lib/solid_objects/actor.rbs index 0f6648f..6915848 100644 --- a/sig/generated/lib/solid_objects/actor.rbs +++ b/sig/generated/lib/solid_objects/actor.rbs @@ -351,6 +351,9 @@ module SolidObjects # @rbs () -> Integer def intent_count: () -> Integer + # @rbs (String) { () -> untyped } -> untyped + def read_projection: (String) { () -> untyped } -> untyped + private attr_reader effect_intents: untyped @@ -363,9 +366,6 @@ module SolidObjects attr_reader outbound_message_intents: untyped - # @rbs (String) { () -> untyped } -> untyped - def read_projection: (String) { () -> untyped } -> untyped - # @rbs () -> Array[Array[Hash[Symbol, untyped]]] def intent_snapshot: () -> Array[Array[Hash[Symbol, untyped]]] diff --git a/sig/generated/lib/solid_objects/actor_snapshot.rbs b/sig/generated/lib/solid_objects/actor_snapshot.rbs index a674f30..d0de883 100644 --- a/sig/generated/lib/solid_objects/actor_snapshot.rbs +++ b/sig/generated/lib/solid_objects/actor_snapshot.rbs @@ -12,6 +12,8 @@ module SolidObjects @revision: Integer + @state_data: Hash[String, untyped] + @observable_values: Hash[String, untyped]? @observable_value_cache: Hash[String, untyped] @@ -35,11 +37,14 @@ module SolidObjects # @rbs (Symbol | String) -> untyped def observable_value: (Symbol | String) -> untyped + # @rbs () -> Actor + def build_actor: () -> Actor + private attr_reader instance: untyped - # @rbs () -> Actor - def build_actor: () -> Actor + # @rbs () -> Hash[String, untyped] + def migrated_state: () -> Hash[String, untyped] end end diff --git a/sig/generated/lib/solid_objects/payload_broadcast.rbs b/sig/generated/lib/solid_objects/payload_broadcast.rbs index 4b9f44f..041daa4 100644 --- a/sig/generated/lib/solid_objects/payload_broadcast.rbs +++ b/sig/generated/lib/solid_objects/payload_broadcast.rbs @@ -2,8 +2,6 @@ module SolidObjects class PayloadBroadcast - MAXIMUM_PAYLOAD_BYTES: ::Integer - REVISION_OBSERVABLE: ::String @snapshot: ActorSnapshot @@ -39,8 +37,8 @@ module SolidObjects # Distinguishes a block that relied on the old class-level receiver from an # ordinary typo, so the one behaviour change reports itself instead of # surfacing as an unexplained NameError. - # @rbs (NameError[untyped]) -> bool - def class_level_receiver?: (NameError[untyped]) -> bool + # @rbs (NameError[untyped], Actor?) -> bool + def class_level_receiver?: (NameError[untyped], Actor?) -> bool # @rbs () -> void def authorize!: () -> void diff --git a/test/integration/payload_projection_test.rb b/test/integration/payload_projection_test.rb new file mode 100644 index 0000000..ebc12e3 --- /dev/null +++ b/test/integration/payload_projection_test.rb @@ -0,0 +1,103 @@ +# rbs_inline: enabled + +require "database_test_helper" + +class PayloadProjectionTest < ActiveSupport::TestCase + class Reader < SolidObjects::Actor + actor_type "payload-projection" + attribute :items, default: [] + attribute :token, default: -> { SecureRandom.uuid } + + broadcast_payload :impure do |_actor, action| + case action + when "state" then items << "unexpected" + when "effect" then emit :unexpected + when "recovery" then request_effect_recovery("effect_id" => "missing-effect") + when "commit_action" then commit_action :unexpected + when "reminder" then schedule(at: Time.now + 60).append + when "outbound" then send_to(self.class.ref("other")).append + when "database" then SolidObjectsTestDomainRecord.create!(name: "unexpected") + when "raise" + items << "unexpected" + raise "projection failed" + end + { "items" => items } + end + + broadcast_payload(:pure) { { "items" => items } } + broadcast_payload(:text) { |_actor, text| { "text" => text } } + broadcast_payload(:defaults) { { "token" => token } } + + def append + items << "committed" + end + end + + setup do + Reader.ensure_registered! + end + + %w[state effect recovery commit_action reminder outbound].each do |action| + test "payload projections reject #{action}" do + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("one")) + + assert_raises(SolidObjects::QueryMutatedState) { render(snapshot, "impure", action) } + + assert_empty snapshot.actor.state.to_h.fetch("items") + assert_equal 0, snapshot.actor.intent_count + end + end + + test "payload projections prevent application database writes" do + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("one")) + + assert_raises(SolidObjects::ApplicationWriteForbidden) { render(snapshot, "impure", "database") } + + assert_empty SolidObjectsTestDomainRecord.all + end + + test "a raising payload cannot contaminate another projection of the same snapshot" do + reference = Reader.ref("one") + reference.append + snapshot = SolidObjects::ActorSnapshot.new(reference) + + assert_raises(RuntimeError) { render(snapshot, "impure", "raise") } + + assert_equal({ "items" => [ "committed" ] }, render(snapshot, "pure").fetch("payload")) + assert_equal [ "committed" ], snapshot.actor.state.to_h.fetch("items") + end + + test "payload projections enforce the configured UTF-8 byte boundary" do + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("one")) + text = "éé" + size = JSON.generate("text" => text).bytesize + SolidObjects.configuration.max_payload_bytes = size + + assert_equal({ "text" => text }, render(snapshot, "text", text).fetch("payload")) + + SolidObjects.configuration.max_payload_bytes = size - 1 + assert_raises(SolidObjects::PayloadTooLarge) { render(snapshot, "text", text) } + end + + test "payload projections share generated defaults from the original snapshot" do + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("one")) + expected = { "token" => snapshot.actor.token } + + assert_equal expected, render(snapshot, "defaults").fetch("payload") + assert_equal expected, render(snapshot, "defaults").fetch("payload") + end + + test "payload projections accept a configured limit above one megabyte" do + snapshot = SolidObjects::ActorSnapshot.new(Reader.ref("one")) + text = "x" * 1_048_576 + SolidObjects.configuration.max_payload_bytes = JSON.generate("text" => text).bytesize + + assert_equal text, render(snapshot, "text", text).fetch("payload").fetch("text") + end + + private + + def render(snapshot, name, authorization_context = nil) + SolidObjects::PayloadBroadcast.new(snapshot:, name:, authorization_context:).call + end +end diff --git a/test/integration/sqlite_busy_wait_test.rb b/test/integration/sqlite_busy_wait_test.rb new file mode 100644 index 0000000..a8d22d7 --- /dev/null +++ b/test/integration/sqlite_busy_wait_test.rb @@ -0,0 +1,53 @@ +# rbs_inline: enabled + +require "database_test_helper" +require "timeout" + +class SqliteBusyWaitTest < ActiveSupport::TestCase + test "background transactions let the SQLite lock holder commit while waiting" do + skip "SQLite busy handlers" unless database_family == :sqlite + + SolidObjectsTestDomainRecord.columns + ready = Queue.new + start = Queue.new + waiting = Queue.new + errors = Queue.new + SolidObjects.configuration.lock_retry_attempts = 0 + adapter = SolidObjects.database_adapter + contender = Thread.new do + SolidObjects::Record.connection_pool.with_connection do |connection| + raw_connection = connection.raw_connection + raw_connection.busy_timeout = 100 + raw_connection.define_singleton_method(:busy_handler) do |&handler| + super() do |count| + waiting << true if count.zero? + handler.call(count) + end + end + ready << true + start.pop + adapter.transaction { SolidObjectsTestDomainRecord.create!(name: "contender") } + rescue => error + errors << error + ensure + waiting << true + if raw_connection && !raw_connection.closed? + raw_connection.singleton_class.remove_method(:busy_handler) + raw_connection.busy_handler_timeout = configured_sqlite_busy_handler_timeout + end + end + end + Timeout.timeout(5) { ready.pop } + SolidObjects::Record.transaction do + SolidObjectsTestDomainRecord.create!(name: "holder") + start << true + Timeout.timeout(5) { waiting.pop } + end + assert contender.join(5), "the contender did not finish after the holder committed" + assert_empty errors.size.times.map { errors.pop } + assert_equal %w[contender holder], SolidObjectsTestDomainRecord.order(:name).pluck(:name) + ensure + start << true if start + contender&.kill&.join if contender&.alive? + end +end diff --git a/test/integration/synchronous_invocation_test.rb b/test/integration/synchronous_invocation_test.rb index cb7558b..f0f231b 100644 --- a/test/integration/synchronous_invocation_test.rb +++ b/test/integration/synchronous_invocation_test.rb @@ -338,6 +338,8 @@ def wait(timeout:) end test "sync does not steal an unexpired activation" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } reference = CounterActor.ref("leased") reference.async.increment instance = SolidObjects::Instance.find_by!( @@ -363,6 +365,10 @@ def wait(timeout:) assert_equal timed_out_message.sequence, error.sequence assert_equal "ready", error.status assert_equal "activation_held", error.waiting_on + attributes = events.find { |event| event.fetch("name") == "solid_objects.sync.timeout" }.fetch("attributes") + assert_equal "activationHeld", attributes.fetch("waitingOn") + assert_equal process_record.id, attributes.fetch("activationOwnerId") + assert_equal lease.generation.to_s, attributes.fetch("activationGeneration") assert_equal process_record.id, error.activation.fetch("owner_id") assert_equal "worker", error.activation.fetch("process").fetch("kind") assert_equal "test-host", error.activation.fetch("process").fetch("hostname") diff --git a/test/integration/telemetry_test.rb b/test/integration/telemetry_test.rb index 498aa73..ca7cf74 100644 --- a/test/integration/telemetry_test.rb +++ b/test/integration/telemetry_test.rb @@ -30,6 +30,32 @@ def increment assert_equal({ "role" => "actors", "previousIntervalMilliseconds" => 100, "currentIntervalMilliseconds" => 200, "reason" => "idle" }, events.last.fetch("attributes")) end + JSON.parse(File.read(File.expand_path("../../compatibility/sync-timeout.json", __dir__))).each do |fixture| + test "portable timeout telemetry preserves #{fixture.fetch("waitingOn")} diagnostics" do + attributes = SolidObjects::Telemetry.event(:"sync.timeout", + waiting_on: fixture.fetch("rubyReason"), + activation_owner_id: "worker-1", + activation_generation: 7, + arguments: { secret: "private" }).fetch("attributes") + + assert_equal({ "waitingOn" => fixture.fetch("waitingOn"), "activationOwnerId" => "worker-1", "activationGeneration" => "7" }, attributes) + end + end + + test "portable database contention telemetry preserves unknown activation fields" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + reference = Counter.ref("contention").async.increment + + error = SolidObjects::SyncDiagnostics.new.database_contention_for(reference, timeout: 1) + + assert_equal "database_contention", error.waiting_on + attributes = events.last.fetch("attributes") + assert_equal "databaseContention", attributes.fetch("waitingOn") + assert_nil attributes.fetch("activationOwnerId") + assert_nil attributes.fetch("activationGeneration") + end + test "a failing started subscriber cannot fail a turn" do subscriber = ActiveSupport::Notifications.subscribe("solid_objects.message.started") { raise "private sink failure" } assert_equal 1, Counter.ref("one").increment From 30bf741b2053e7d416a1698253990d08a330ff6e Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Thu, 1 Oct 2026 07:36:33 -0700 Subject: [PATCH 07/11] fix: Share the portable event contract A parity audit found portable events with different names and fields in Ruby and JavaScript. Only the timeout event had a shared fixture, so no test found the difference. compatibility/telemetry-events.json now holds the attribute allowlist and the exact keys of each core SQL event. Both test suites read the same file. - Rename payload_broadcast_failed to payload_broadcast.failed, the dotted form that all other events use. - Send operation, deliveryMode, retryable, outcome, commitAction, and the outbox identity, as JavaScript already does. - Log exporter and observer failures. Ruby discarded them before. - Require an observer block and accept at most 1,000 observers. - Pin reserved JSON keys through actor arguments, state, and results. --- CHANGELOG.md | 16 + compatibility/telemetry-events.json | 378 ++++++++++++++++++ compatibility/transmit-envelopes.json | 2 +- docs/authorization.md | 7 +- docs/observability.md | 75 ++-- docs/realtime.md | 7 +- docs/roadmap.md | 4 +- lib/solid_objects.rb | 8 + lib/solid_objects/activation.rb | 4 +- lib/solid_objects/actor_channel.rb | 4 +- lib/solid_objects/client.rb | 3 +- lib/solid_objects/diagnostics.rb | 11 +- lib/solid_objects/executor.rb | 11 +- lib/solid_objects/mailbox.rb | 4 +- lib/solid_objects/observer_registry.rb | 47 +++ lib/solid_objects/reference.rb | 2 + lib/solid_objects/reminder_scheduler.rb | 1 + lib/solid_objects/telemetry.rb | 59 ++- sig/generated/lib/solid_objects.rbs | 3 + .../lib/solid_objects/observer_registry.rbs | 29 ++ sig/generated/lib/solid_objects/telemetry.rbs | 9 + sig/support/framework.rbs | 5 + test/integration/actor_channel_test.rb | 5 +- test/integration/json_compatibility_test.rb | 34 ++ test/integration/payload_delivery_test.rb | 17 +- test/integration/process_lifecycle_test.rb | 4 + .../synchronous_invocation_test.rb | 8 + test/integration/telemetry_test.rb | 162 +++++++- test/portable_telemetry_assertions.rb | 25 ++ test/unit/serialization_test.rb | 6 + 30 files changed, 886 insertions(+), 64 deletions(-) create mode 100644 compatibility/telemetry-events.json create mode 100644 lib/solid_objects/observer_registry.rb create mode 100644 sig/generated/lib/solid_objects/observer_registry.rbs create mode 100644 test/integration/json_compatibility_test.rb create mode 100644 test/portable_telemetry_assertions.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index d55f3c5..7317419 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,22 @@ ## Unreleased +- Rename `solid_objects.payload_broadcast_failed` to + `solid_objects.payload_broadcast.failed`, the dotted form that every other + event uses. Update Active Support subscribers to the new name. The portable + event names the payload `payload`. +- Match portable event attributes to JavaScript through the shared + `compatibility/telemetry-events.json` contract. Message events carry + `operation` and `deliveryMode`, `message.failed` carries `retryable` and + `outcome`, commit action events carry the message fields and `commitAction`, + and `reminder.enqueued` carries `operation`. `outbox.age` carries the effect or + broadcast identity, `sync.enqueue_timeout` carries `timeoutMilliseconds`, and + polling intervals are integers. `realtime.connected` carries only actor fields. +- Log `solid_objects.instrumentation.failed` when an exporter or observer raises. + Observers require a block, a process accepts at most 1,000 observers, and + `SolidObjects.reset!` removes them. Pin reserved JSON keys through actor + arguments, state, and retained results. + - Guard personalized payload projections against state changes, staged work, and application database writes. Each payload gets an isolated actor from the committed snapshot and honors `max_payload_bytes`, matching JavaScript. diff --git a/compatibility/telemetry-events.json b/compatibility/telemetry-events.json new file mode 100644 index 0000000..19669db --- /dev/null +++ b/compatibility/telemetry-events.json @@ -0,0 +1,378 @@ +{ + "attributes": [ + "activationGeneration", + "activationOwnerId", + "actorId", + "actorType", + "ageMilliseconds", + "attempt", + "broadcastId", + "broadcastWorkers", + "byteCount", + "code", + "commitAction", + "component", + "componentCount", + "count", + "currentIntervalMilliseconds", + "deliveryMode", + "depth", + "durationMilliseconds", + "effectId", + "effectName", + "effectWorkers", + "errorName", + "failureCount", + "generation", + "idlePollingIntervalMilliseconds", + "incarnation", + "instanceId", + "intervalMilliseconds", + "latenessMilliseconds", + "messageId", + "name", + "nextRunAt", + "occurrence", + "operation", + "outboxKind", + "outcome", + "ownerId", + "payload", + "phase", + "pollingIntervalMilliseconds", + "previousIntervalMilliseconds", + "previousRunAt", + "processId", + "processKind", + "reason", + "reminderId", + "reminderSchedulers", + "requestId", + "retryable", + "revision", + "role", + "sequence", + "status", + "thresholdBytes", + "timeoutMilliseconds", + "truncated", + "waitingOn", + "workers" + ], + "events": [ + { + "name": "activation.completed", + "attributes": ["actorId", "actorType", "generation", "instanceId", "ownerId"], + "javascriptAttributes": [ + "attempt", + "deliveryMode", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "activation.failed", + "attributes": ["actorId", "actorType", "errorName", "generation", "instanceId", "ownerId"], + "javascriptAttributes": [ + "attempt", + "deliveryMode", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "activation.started", + "attributes": ["actorId", "actorType", "generation", "instanceId", "ownerId"], + "javascriptAttributes": [ + "attempt", + "deliveryMode", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "commit_action.completed", + "attributes": [ + "activationGeneration", + "actorId", + "actorType", + "attempt", + "commitAction", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "commit_action.failed", + "attributes": [ + "activationGeneration", + "actorId", + "actorType", + "attempt", + "commitAction", + "deliveryMode", + "errorName", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "commit_action.started", + "attributes": [ + "activationGeneration", + "actorId", + "actorType", + "attempt", + "commitAction", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "dead_letter.created", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "mailbox.depth", + "attributes": ["actorId", "actorType", "count", "depth", "instanceId", "truncated"] + }, + { + "name": "message.completed", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "durationMilliseconds", + "instanceId", + "messageId", + "operation", + "requestId", + "revision", + "sequence" + ] + }, + { + "name": "message.enqueued", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "message.failed", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "durationMilliseconds", + "errorName", + "instanceId", + "messageId", + "operation", + "outcome", + "requestId", + "retryable", + "sequence" + ] + }, + { + "name": "message.rejected", + "attributes": [ + "actorId", + "actorType", + "attempt", + "code", + "deliveryMode", + "durationMilliseconds", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "message.retry", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "message.started", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "outbox.age", + "match": { + "outboxKind": "broadcast" + }, + "attributes": [ + "actorId", + "actorType", + "ageMilliseconds", + "attempt", + "broadcastId", + "instanceId", + "messageId", + "outboxKind", + "revision" + ] + }, + { + "name": "outbox.age", + "match": { + "outboxKind": "effect" + }, + "attributes": [ + "actorId", + "actorType", + "ageMilliseconds", + "attempt", + "effectId", + "effectName", + "instanceId", + "messageId", + "outboxKind" + ] + }, + { + "name": "payload_broadcast.failed", + "attributes": ["actorId", "actorType", "errorName", "payload"] + }, + { + "name": "polling.interval_changed", + "attributes": [ + "currentIntervalMilliseconds", + "previousIntervalMilliseconds", + "reason", + "role" + ] + }, + { + "name": "realtime.connected", + "attributes": ["actorId", "actorType"] + }, + { + "name": "realtime.disconnected", + "attributes": ["actorId", "actorType"] + }, + { + "name": "recovery.completed", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "recovery.failed", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "recovery.reclaimed", + "attributes": [ + "actorId", + "actorType", + "attempt", + "deliveryMode", + "instanceId", + "messageId", + "operation", + "requestId", + "sequence" + ] + }, + { + "name": "reminder.enqueued", + "attributes": [ + "actorId", + "actorType", + "attempt", + "instanceId", + "latenessMilliseconds", + "messageId", + "occurrence", + "operation", + "reminderId" + ] + }, + { + "name": "snapshot.read", + "attributes": ["actorId", "actorType", "instanceId", "revision"] + }, + { + "name": "sync.enqueue_timeout", + "attributes": ["actorId", "actorType", "operation", "timeoutMilliseconds"] + } + ] +} diff --git a/compatibility/transmit-envelopes.json b/compatibility/transmit-envelopes.json index a1b9249..c47137a 100644 --- a/compatibility/transmit-envelopes.json +++ b/compatibility/transmit-envelopes.json @@ -82,7 +82,7 @@ "actorType": "transmit-counters", "actorId": "fixture-counter", "operation": "increment", - "arguments": [ 1 ] + "arguments": [1] } }, { diff --git a/docs/authorization.md b/docs/authorization.md index 468bea8..c74290d 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -13,7 +13,7 @@ answers nothing until the host application defines its trust boundary. | `authorize_query` | Attribute reads, declared queries, committed snapshots, scalar observable reads, initial component rendering, and every component refresh dependency | Explicit call context, the context passed to `solid_object`, or the request context resolved for a component refresh | Actor state or personalized projections can leak across users or tenants | | `authorize_destroy` | `reference.destroy` | Value passed as `authorization_context:` | Complete actor state, mailbox, reminders, and pending outboxes can be deleted | | `authorize_subscription` | Action Cable subscription to one actor stream | The `ActionCable::Connection` object | Clients can receive future observable updates for other actors | -| `authorize_administration` | Engine administration controllers, every `SolidObjects::Web` page, process inspection/cleanup/pruning, message pruning, and dead-letter inspection/retry | Rails controller, a `SolidObjects::Web` request that answers `request`/`session`/`env`, or `{ source: "cli" }` | Operational metadata, arguments, errors, deletion, and retries become exposed or mutable | +| `authorize_administration` | Engine administration controllers, every `SolidObjects::Web` page, process inspection/cleanup/pruning, message pruning, dead-letter inspection/retry, and actor diagnostics and observers | Rails controller, a `SolidObjects::Web` request that answers `request`/`session`/`env`, or `{ source: "cli" }` | Operational metadata, arguments, errors, deletion, and retries become exposed or mutable | Every `MessageReference#status`, `#result`, `#outcome`, and `#wait` call reauthorizes the stored operation and arguments as a message or query. Supply @@ -22,6 +22,11 @@ does not retain the original caller's context. Internal reminder, effect-callbac actor-to-actor deliveries come from already committed runtime rows and do not re-enter the public client policy. +Actor diagnostics and actor observers call `authorize_administration` with the +resource `actor_diagnostics` and the resource ID `[actor_type, actor_id].to_json`. +`diagnostics` uses the action `:inspect`. `observe` and `on` use the action +`:observe`. The check runs before any queue read or observer registration. + ## Realtime authorization contexts Reactive components cross three Rails execution contexts and authorize at all diff --git a/docs/observability.md b/docs/observability.md index 4e893b6..feae949 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -22,27 +22,49 @@ attributes are excluded. Actor IDs remain correlation data: applications should use opaque actor identifiers and apply their own retention policy to event logs. Events and metric samples are immutable. Throwing observers, rejected observer promises, and a failing instrumentation error logger cannot change a turn's -result. Delivery is best effort and synchronous callbacks should be short; +result. When an exporter or observer raises, both runtimes log +`solid_objects.instrumentation.failed` with the event name and the error class. +Delivery is best effort and synchronous callbacks should be short; JavaScript does not await exporters. Telemetry is not a durable audit trail. -| Event | Meaning | -| --- | --- | -| `activation.started/completed/failed` | Local actor activation hook lifecycle | -| `message.started/completed/failed/rejected` | One attempt's execution outcome | -| `message.retry` | Failed attempt durably queued for another attempt | -| `dead_letter.created` | Message exhausted retries or failed permanently | -| `mailbox.depth` | On-demand diagnostic sample; exact depth only if not truncated | -| `reminder.enqueued` | Due reminder dispatch; lateness is measured from due time | -| `outbox.age` | Delivery observation; age is time since the item's current availability time | -| `recovery.reclaimed` | A previously claimed, interrupted message begins another attempt | -| `recovery.completed/failed` | Durable effect recovery callback commits or enters the dead-letter queue | -| `snapshot.read` | Authorized snapshot constructed without exposing its contents | -| `realtime.connected/disconnected` | Actor subscription added or removed | +| Event | Meaning | +| ------------------------------------------- | ---------------------------------------------------------------------------- | +| `activation.started/completed/failed` | Local actor activation hook lifecycle | +| `message.started/completed/failed/rejected` | One attempt's execution outcome | +| `message.retry` | Failed attempt durably queued for another attempt | +| `dead_letter.created` | Message exhausted retries or failed permanently | +| `commit_action.started/completed/failed` | One registered commit action inside the commit transaction | +| `mailbox.depth` | On-demand diagnostic sample; `depth` is null when the sample is truncated | +| `reminder.enqueued` | Due reminder dispatch; lateness is measured from due time | +| `outbox.age` | Delivery observation; age is time since the item's current availability time | +| `recovery.reclaimed` | A previously claimed, interrupted message begins another attempt | +| `recovery.completed/failed` | Durable effect recovery callback commits or enters the dead-letter queue | +| `snapshot.read` | Authorized snapshot constructed without exposing its contents | +| `realtime.connected/disconnected` | Actor subscription added or removed | +| `payload_broadcast.failed` | One personalized payload failed; `payload` names it | Events describe local observations. Concurrent deletion, crashes, and failed exporters can omit events. Never infer exactly-once delivery from event counts. Additional existing runtime events retain their names. +## Event attributes + +`compatibility/telemetry-events.json` holds the attribute allowlist and the exact +attribute keys of each core event. Both test suites compare the events of the SQL +runtimes with this file. Message events carry `operation` and `deliveryMode`. +`message.failed` also carries a boolean `retryable` and an `outcome` of +`retrying` or `dead`. Commit action events carry `commitAction` and +`activationGeneration`. Activation events carry `generation` and `ownerId`. + +Ruby activates an actor instance before it claims a message, so its activation +events have no message fields. JavaScript activates an actor in the turn that +claims a message, so its activation events also carry the `messageId`, +`requestId`, `sequence`, `attempt`, `operation`, and `deliveryMode` of that +message. The contract file records these keys as JavaScript-only. + +The Durable Objects host sends the same envelope, but some of its events carry +fewer attributes. The contract file does not apply to that host. + Portable `polling.interval_changed` events carry `previousIntervalMilliseconds`, `currentIntervalMilliseconds`, and a string `reason`. Ruby converts its native second-based notification values while preserving the original notification. @@ -56,20 +78,21 @@ unavailable activation fields are null. Both runtimes use the same wait reasons: `notYetAvailable`, `readyUnclaimed`, `databaseContention`, and `unknown`. Ruby's exception attributes and Active Support notifications retain their native snake_case names and reason values. The portable instrumentation envelope uses -the shared camelCase contract. +the shared camelCase contract. The Durable Objects host does not send +`sync.timeout`. A `call` timeout there reports `waitingOn: "unknown"`. ## Metrics and tracing Metrics are sample descriptions. Exporting them is opt-in: the runtime does not register meters, allocate per-actor metric series, or install a vendor SDK. -| Name | Kind | Unit | Aggregation | -| --- | --- | --- | --- | -| `solid_objects.events` | counter | `1` | Sum one per event | -| `solid_objects.duration` | histogram | `ms` | Distribution of observed attempt duration | -| `solid_objects.reminder.lateness` | histogram | `ms` | Distribution of reminder dispatch delay | -| `solid_objects.outbox.age` | histogram | `ms` | Distribution of delivery delay since availability | -| `solid_objects.mailbox.depth` | gauge | `1` | Last exact sampled actor depth; omit truncated samples | +| Name | Kind | Unit | Aggregation | +| --------------------------------- | --------- | ---- | ------------------------------------------------------ | +| `solid_objects.events` | counter | `1` | Sum one per event | +| `solid_objects.duration` | histogram | `ms` | Distribution of observed attempt duration | +| `solid_objects.reminder.lateness` | histogram | `ms` | Distribution of reminder dispatch delay | +| `solid_objects.outbox.age` | histogram | `ms` | Distribution of delivery delay since availability | +| `solid_objects.mailbox.depth` | gauge | `1` | Last exact sampled actor depth; omit truncated samples | Labels contain only event name, adapter family, and declared actor type. Keep the actor type registry finite. Never add actor ID, incarnation, message ID, operation @@ -94,7 +117,8 @@ Ruby uses `reference.observe(authorization_context:) { |event| ... }` and calling the returned proc. `on` filters one event name, such as `message.retry`. Observers receive only this actor's events in the current runtime/process; they are not subscriptions to workers on other hosts. Dispose them when the caller's -session ends or authorization is revoked. JS limits local observers to 1,000. +session ends or authorization is revoked. Each runtime or process accepts at most +1,000 local observers. An observer needs a callback or a block. For remote Durable Objects, configure `instrumentation` on the actor host and filter by actor identity there; process-local reference observers raise `UnsupportedCapability`. Remote `reference.diagnostics` is supported. @@ -105,8 +129,9 @@ to allow action `observe` or `inspect`, resource `actor_diagnostics`, and resour runs before reading summaries or registering observers; possessing an actor ID confers no permission. -Diagnostics read at most `limit + 1` rows per queue source, with a hard limit of -100. Each category returns `sampled`, `truncated`, and `oldestAgeMilliseconds`. +Diagnostics read at most `limit + 1` rows per queue source, with a hard limit +of 100 rows. Each category returns `sampled`, `truncated`, and +`oldestAgeMilliseconds`. The limit applies to each combined category: one effect plus one broadcast with `limit: 1` returns `sampled: 1, truncated: true`, even when both source queries returned all their rows. The extra row proves that the category exceeds its cap. diff --git a/docs/realtime.md b/docs/realtime.md index 04eb011..f36e5e4 100644 --- a/docs/realtime.md +++ b/docs/realtime.md @@ -347,8 +347,9 @@ object or array so the wire format stays inspectable. A payload is one subscriber's view of one name, so a failure is confined to it. A raising block does not reject the subscription, stop the other payload names, or stop component refreshes on the same connection. The failure is reported as -`solid_objects.payload_broadcast_failed` carrying the actor type, actor id, -payload name, and exception class. The exception message is deliberately not +`solid_objects.payload_broadcast.failed` carrying the actor type, actor id, +payload name, and exception class. The Active Support payload names the payload +`payload_name`; the portable instrumentation event names it `payload`. The exception message is deliberately not included: a payload block reads subscriber state, so its message is the one place that state could leak into logs. @@ -356,7 +357,7 @@ A revision with a failed payload does not advance the delivery watermark, so a transient failure is retried on the next broadcast rather than being recorded as delivered. Retries are driven by broadcasts rather than a timer, so a payload that fails persistently retries once per actor mutation and reports each -attempt. A repeating stream of `payload_broadcast_failed` for one `payload_name` +attempt. A repeating stream of `payload_broadcast.failed` for one `payload_name` therefore means a persistent fault in that block, not a one-off; a single event that does not recur was transient and has already been recovered. diff --git a/docs/roadmap.md b/docs/roadmap.md index 0e424f5..75cb615 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,7 +4,9 @@ - Portable telemetry with a shared JSON schema, metric samples, isolated observer callbacks, bounded authorized actor diagnostics, and matching timeout wait - reasons and activation metadata. See [observability](observability.md). + reasons and activation metadata. Both test suites check the attribute keys of + each core SQL event against `compatibility/telemetry-events.json`. See + [observability](observability.md). - Rails engine, install generator, migration, and CLI - Explicit actor registry, references, JSON state, and state migrations diff --git a/lib/solid_objects.rb b/lib/solid_objects.rb index 9c84640..d6259a5 100644 --- a/lib/solid_objects.rb +++ b/lib/solid_objects.rb @@ -69,6 +69,7 @@ require "solid_objects/effect_recovery_coordinator" require "solid_objects/process_heartbeat" require "solid_objects/commit_action_registry" +require "solid_objects/observer_registry" require "solid_objects/lease" require "solid_objects/lease_renewer" # The reminder scheduler and the effect executor enqueue through the mailbox, @@ -141,6 +142,11 @@ def register_commit_action(name, &handler) commit_action_registry.register(name, handler) end + # @rbs () -> ObserverRegistry + def observer_registry + @observer_registry ||= ObserverRegistry.new + end + # @rbs () -> Client def client require "solid_objects/client" @@ -205,6 +211,8 @@ def reset! @caller_process = nil @effect_registry = EffectRegistry.new @commit_action_registry = CommitActionRegistry.new + @observer_registry&.clear + @observer_registry = ObserverRegistry.new @dead_letters = nil @redrives = nil @administration = nil diff --git a/lib/solid_objects/activation.rb b/lib/solid_objects/activation.rb index c7ba0d9..a87b243 100644 --- a/lib/solid_objects/activation.rb +++ b/lib/solid_objects/activation.rb @@ -24,7 +24,7 @@ def initialize(lease:) @actor = build_actor(instance) @last_used_at = monotonic_now @pass_exhausted = false - SolidObjects.instrument(:"activation.started", instance_id: instance.id, actor_type: instance.actor_type, actor_id: instance.actor_id, generation: lease.generation) + SolidObjects.instrument(:"activation.started", instance_id: instance.id, actor_type: instance.actor_type, actor_id: instance.actor_id, owner_id: lease.owner_id, generation: lease.generation) actor.activate SolidObjects.instrument( :"activation.completed", @@ -35,7 +35,7 @@ def initialize(lease:) generation: lease.generation ) rescue => error - SolidObjects.instrument(:"activation.failed", instance_id: lease.instance_id, actor_type: instance&.actor_type, actor_id: instance&.actor_id, error_class: error.class.name) + SolidObjects.instrument(:"activation.failed", instance_id: lease.instance_id, actor_type: instance&.actor_type, actor_id: instance&.actor_id, owner_id: lease.owner_id, generation: lease.generation, error_class: error.class.name) raise end diff --git a/lib/solid_objects/actor_channel.rb b/lib/solid_objects/actor_channel.rb index 71798a0..af3c75a 100644 --- a/lib/solid_objects/actor_channel.rb +++ b/lib/solid_objects/actor_channel.rb @@ -49,7 +49,7 @@ def subscribed end refresh_outdated_components(snapshot) transmit_state_payloads(snapshot) - SolidObjects.instrument(:"realtime.connected", actor_type: reference.actor_type, actor_id: reference.actor_id, instance_id: snapshot.instance_id, revision: snapshot.revision) + SolidObjects.instrument(:"realtime.connected", actor_type: reference.actor_type, actor_id: reference.actor_id) rescue KeyError, JSON::ParserError, InvalidStreamToken, @@ -149,7 +149,7 @@ def transmit_state_payload(snapshot, name) true rescue => error SolidObjects.instrument( - :payload_broadcast_failed, + :"payload_broadcast.failed", actor_type: reference.actor_type, actor_id: reference.actor_id, payload_name: name, diff --git a/lib/solid_objects/client.rb b/lib/solid_objects/client.rb index c15d90d..3dd4c69 100644 --- a/lib/solid_objects/client.rb +++ b/lib/solid_objects/client.rb @@ -206,7 +206,8 @@ def enqueue_sync(reference:, operation:, arguments:, idempotency_key:, timeout:) :"sync.enqueue_timeout", actor_type: reference.actor_type, actor_id: reference.actor_id, - operation: operation.to_s + operation: operation.to_s, + timeout_milliseconds: (timeout * 1000).round ) raise SyncEnqueueTimeout.new( timeout:, diff --git a/lib/solid_objects/diagnostics.rb b/lib/solid_objects/diagnostics.rb index 8a837ed..71d9c6c 100644 --- a/lib/solid_objects/diagnostics.rb +++ b/lib/solid_objects/diagnostics.rb @@ -44,18 +44,15 @@ def summary(limit: 100, authorization_context: nil) # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc def observe(authorization_context: nil, &block) + raise ArgumentError, "an actor observer requires a block" unless block + authorize!(:observe, authorization_context:) - subscription = ActiveSupport::Notifications.subscribe(/\Asolid_objects\./) do |notification| + SolidObjects.observer_registry.subscribe do |notification| payload = notification.payload next unless payload[:actor_type] == reference.actor_type && payload[:actor_id] == reference.actor_id - begin - block.call(Telemetry.event(notification.name.delete_prefix("solid_objects.").to_sym, payload)) - rescue - nil - end + Telemetry.deliver(block, Telemetry.event(notification.name.delete_prefix("solid_objects.").to_sym, payload)) end - -> { ActiveSupport::Notifications.unsubscribe(subscription) } end private diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index 060937e..10adce1 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -212,11 +212,8 @@ def execute_commit_actions(intents) # @rbs (intent: Actor::CommitActionIntent, handler: Proc, context: CommitActionContext) -> untyped def execute_commit_action(intent:, handler:, context:) payload = { + **instrumentation_payload, commit_action_name: intent.name, - message_id: message.id, - request_id: message.request_id, - actor_type: message.actor_type, - actor_id: message.actor_id, activation_generation: activation.lease.generation } SolidObjects.instrument(:"commit_action.started", **payload) @@ -424,6 +421,8 @@ def fail_message(error) **instrumentation_payload, error_class: error.class.name, duration_milliseconds: elapsed_milliseconds, + retryable: !error.is_a?(NonRetryableError), + outcome: dead ? "dead" : "retrying", dead: ) SolidObjects.instrument_after_commit(:"recovery.failed", **instrumentation_payload) if dead && recovery_message? @@ -545,7 +544,9 @@ def instrumentation_payload actor_id: message.actor_id, sequence: message.sequence, attempt: message.attempt_count, - request_id: message.request_id + request_id: message.request_id, + operation: message.operation, + delivery_mode: message.delivery_mode } end end diff --git a/lib/solid_objects/mailbox.rb b/lib/solid_objects/mailbox.rb index 0822816..0c4ebfb 100644 --- a/lib/solid_objects/mailbox.rb +++ b/lib/solid_objects/mailbox.rb @@ -85,7 +85,9 @@ def announce(message) actor_type: message.actor_type, actor_id: message.actor_id, sequence: message.sequence, - request_id: message.request_id + request_id: message.request_id, + operation: message.operation, + delivery_mode: message.delivery_mode ) SolidObjects.wake_up.signal end diff --git a/lib/solid_objects/observer_registry.rb b/lib/solid_objects/observer_registry.rb new file mode 100644 index 0000000..1e8339c --- /dev/null +++ b/lib/solid_objects/observer_registry.rb @@ -0,0 +1,47 @@ +# rbs_inline: enabled + +module SolidObjects + class ObserverRegistry + MAXIMUM_OBSERVERS = 1000 + + # @rbs @subscriptions: Array[Object] + # @rbs @mutex: Mutex + + # @rbs () -> void + def initialize + @subscriptions = [] + @mutex = Mutex.new + end + + # @rbs () { (ActiveSupport::Notifications::Event) -> void } -> Proc + def subscribe(&listener) + subscription = mutex.synchronize do + raise ArgumentError, "at most #{MAXIMUM_OBSERVERS} local observers may be registered" if subscriptions.length >= MAXIMUM_OBSERVERS + + ActiveSupport::Notifications.subscribe(/\Asolid_objects\./, &listener).tap { |created| subscriptions << created } + end + -> { unsubscribe(subscription) } + end + + # @rbs () -> void + def clear + mutex.synchronize do + subscriptions.each { |subscription| ActiveSupport::Notifications.unsubscribe(subscription) } + subscriptions.clear + end + end + + private + + attr_reader :subscriptions #: Array[Object] + attr_reader :mutex #: Mutex + + # @rbs (Object) -> void + def unsubscribe(subscription) + mutex.synchronize do + ActiveSupport::Notifications.unsubscribe(subscription) + subscriptions.delete(subscription) + end + end + end +end diff --git a/lib/solid_objects/reference.rb b/lib/solid_objects/reference.rb index 2f6aa09..e92bd28 100644 --- a/lib/solid_objects/reference.rb +++ b/lib/solid_objects/reference.rb @@ -28,6 +28,8 @@ def observe(authorization_context: nil, &block) # @rbs (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc def on(name, authorization_context: nil, &block) + raise ArgumentError, "an actor observer requires a block" unless block + observe(authorization_context:) do |event| block.call(event) if event.fetch("name") == "solid_objects.#{name}" end diff --git a/lib/solid_objects/reminder_scheduler.rb b/lib/solid_objects/reminder_scheduler.rb index 4e635de..dddd7dd 100644 --- a/lib/solid_objects/reminder_scheduler.rb +++ b/lib/solid_objects/reminder_scheduler.rb @@ -173,6 +173,7 @@ def enqueue(reminder, now:) reminder_id: reminder.id, actor_type: reminder.actor_type, actor_id: reminder.actor_id, + operation: reminder.operation, occurrence: reminder.occurrence ) MessageReference.from_message(message) diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb index daa305b..7e2a140 100644 --- a/lib/solid_objects/telemetry.rb +++ b/lib/solid_objects/telemetry.rb @@ -3,15 +3,22 @@ module SolidObjects module Telemetry FIELDS = %w[ - actorType actorId instanceId incarnation revision messageId requestId attempt sequence - operation deliveryMode generation durationMilliseconds latenessMilliseconds ageMilliseconds - depth count errorName outcome retryable status effectId effectName reminderId occurrence - outboxKind truncated broadcastId code role reason processId processKind ownerId componentCount byteCount - thresholdBytes previousRunAt nextRunAt name commitAction payload - previousIntervalMilliseconds currentIntervalMilliseconds - waitingOn activationOwnerId activationGeneration + activationGeneration activationOwnerId actorId actorType ageMilliseconds attempt broadcastId + broadcastWorkers byteCount code commitAction component componentCount count currentIntervalMilliseconds + deliveryMode depth durationMilliseconds effectId effectName effectWorkers errorName failureCount + generation idlePollingIntervalMilliseconds incarnation instanceId intervalMilliseconds + latenessMilliseconds messageId name nextRunAt occurrence operation outboxKind outcome ownerId payload + phase pollingIntervalMilliseconds previousIntervalMilliseconds previousRunAt processId processKind + reason reminderId reminderSchedulers requestId retryable revision role sequence status thresholdBytes + timeoutMilliseconds truncated waitingOn workers ].freeze - ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds" }.freeze + ALIASES = { + "errorClass" => "errorName", + "stateRevision" => "revision", + "durationMs" => "durationMilliseconds", + "commitActionName" => "commitAction", + "payloadName" => "payload" + }.freeze class << self # @rbs (Effect | Broadcast) -> void @@ -27,7 +34,8 @@ def outbox(record) message_id: record.message_id, attempt: record.attempt_count, outbox_kind: record.is_a?(Effect) ? "effect" : "broadcast", - age_milliseconds: [ ((SolidObjects.database_adapter.database_now - record.available_at) * 1000).round, 0 ].max + age_milliseconds: [ ((SolidObjects.database_adapter.database_now - record.available_at) * 1000).round, 0 ].max, + **outbox_identity(record) ) rescue nil @@ -38,11 +46,18 @@ def emit(name, payload) observer = SolidObjects.configuration.instrumentation return unless observer - observer.call(event(name, payload)) + deliver(observer, event(name, payload)) rescue nil end + # @rbs (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void + def deliver(observer, event) + observer.call(event) + rescue => error + log_delivery_failure(event.fetch("name"), error) + end + # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] def event(name, payload) attributes = safe_attributes(payload) @@ -82,7 +97,7 @@ def safe_attributes(payload) payload.each_with_object({}) do |(key, value), attributes| value = value.to_s if key == :reason && value.is_a?(Symbol) if %i[previous_interval current_interval].include?(key) && value.is_a?(Numeric) - attributes["#{key.to_s.camelize(:lower)}Milliseconds"] = value * 1000 + attributes["#{key.to_s.camelize(:lower)}Milliseconds"] = (value * 1000).round next end name = key.to_s.camelize(:lower) @@ -95,6 +110,28 @@ def safe_attributes(payload) attributes[name] = value end end + + private + + # @rbs (Effect | Broadcast) -> Hash[Symbol, String | Integer] + def outbox_identity(record) + return { effect_id: record.effect_id, effect_name: record.name } if record.is_a?(Effect) + + { broadcast_id: record.broadcast_id, revision: record.message.sequence } + end + + # @rbs (String, Exception) -> void + def log_delivery_failure(event_name, error) + SolidObjects.configuration.logger.error( + { + event: "solid_objects.instrumentation.failed", + instrumentation_event: event_name, + error_class: error.class.name + } + ) + rescue + nil + end end end end diff --git a/sig/generated/lib/solid_objects.rbs b/sig/generated/lib/solid_objects.rbs index 0d7563d..1d35d2f 100644 --- a/sig/generated/lib/solid_objects.rbs +++ b/sig/generated/lib/solid_objects.rbs @@ -27,6 +27,9 @@ module SolidObjects # @rbs (String | Symbol) { (Hash[String, untyped], CommitActionContext) -> untyped } -> Proc def self.register_commit_action: (String | Symbol) { (Hash[String, untyped], CommitActionContext) -> untyped } -> Proc + # @rbs () -> ObserverRegistry + def self.observer_registry: () -> ObserverRegistry + # @rbs () -> Client def self.client: () -> Client diff --git a/sig/generated/lib/solid_objects/observer_registry.rbs b/sig/generated/lib/solid_objects/observer_registry.rbs new file mode 100644 index 0000000..e381f64 --- /dev/null +++ b/sig/generated/lib/solid_objects/observer_registry.rbs @@ -0,0 +1,29 @@ +# Generated from lib/solid_objects/observer_registry.rb with RBS::Inline + +module SolidObjects + class ObserverRegistry + MAXIMUM_OBSERVERS: ::Integer + + @subscriptions: Array[Object] + + @mutex: Mutex + + # @rbs () -> void + def initialize: () -> void + + # @rbs () { (ActiveSupport::Notifications::Event) -> void } -> Proc + def subscribe: () { (ActiveSupport::Notifications::Event) -> void } -> Proc + + # @rbs () -> void + def clear: () -> void + + private + + attr_reader subscriptions: Array[Object] + + attr_reader mutex: Mutex + + # @rbs (Object) -> void + def unsubscribe: (Object) -> void + end +end diff --git a/sig/generated/lib/solid_objects/telemetry.rbs b/sig/generated/lib/solid_objects/telemetry.rbs index 08e4b61..f080d3c 100644 --- a/sig/generated/lib/solid_objects/telemetry.rbs +++ b/sig/generated/lib/solid_objects/telemetry.rbs @@ -12,10 +12,19 @@ module SolidObjects # @rbs (Symbol, Hash[Symbol, untyped]) -> void def self.emit: (Symbol, Hash[Symbol, untyped]) -> void + # @rbs (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void + def self.deliver: (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void + # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] def self.event: (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] def self.safe_attributes: (Hash[Symbol, untyped]) -> Hash[String, untyped] + + # @rbs (Effect | Broadcast) -> Hash[Symbol, String | Integer] + private def self.outbox_identity: (Effect | Broadcast) -> Hash[Symbol, String | Integer] + + # @rbs (String, Exception) -> void + private def self.log_delivery_failure: (String, Exception) -> void end end diff --git a/sig/support/framework.rbs b/sig/support/framework.rbs index 7332a4e..4b1a804 100644 --- a/sig/support/framework.rbs +++ b/sig/support/framework.rbs @@ -9,6 +9,11 @@ end module ActiveSupport class MessageVerifier end + + module Notifications + class Event + end + end end module ActionCable diff --git a/test/integration/actor_channel_test.rb b/test/integration/actor_channel_test.rb index 68843b2..e4b740a 100644 --- a/test/integration/actor_channel_test.rb +++ b/test/integration/actor_channel_test.rb @@ -7,11 +7,13 @@ require "action_view/testing/resolvers" require "cgi/escape" require_relative "../../app/helpers/solid_objects/actor_helper" +require "portable_telemetry_assertions" ActionCable.server.config.cable = { "adapter" => "test" } class ActorChannelTest < ActionCable::Channel::TestCase tests SolidObjects::ActorChannel + include PortableTelemetryAssertions class ChannelActor < SolidObjects::Actor actor_type "channel-actor" @@ -99,8 +101,7 @@ def stream_from(_broadcasting, callback = nil, coder: nil, &block) assert_equal 2, updates.length assert_includes updates.last, ">1" unsubscribe - assert_includes events.map { |event| event.fetch("name") }, "solid_objects.realtime.connected" - assert_includes events.map { |event| event.fetch("name") }, "solid_objects.realtime.disconnected" + assert_portable_events(events, %w[realtime.connected realtime.disconnected]) ensure worker&.stop end diff --git a/test/integration/json_compatibility_test.rb b/test/integration/json_compatibility_test.rb new file mode 100644 index 0000000..d550636 --- /dev/null +++ b/test/integration/json_compatibility_test.rb @@ -0,0 +1,34 @@ +# frozen_string_literal: true + +require "database_test_helper" + +class JsonCompatibilityTest < ActiveSupport::TestCase + FIXTURES = JSON.parse(File.read(File.expand_path("../../compatibility/json-values.json", __dir__))).freeze + + class JsonActor < SolidObjects::Actor + actor_type "json-compatibility" + + attribute :payload, default: -> { {} } + + def store(payload:) + self.payload = payload + payload + end + end + + test "preserves reserved keys through actor arguments, state, and results" do + reference = JsonActor.ref("one") + worker = SolidObjects::Worker.new + + FIXTURES.each do |fixture| + value = fixture.fetch("value") + message = reference.async(idempotency_key: fixture.fetch("name")).store(payload: value) + worker.run_until_idle + + assert_equal value, message.result + assert_equal value, reference.snapshot.payload + end + ensure + worker&.stop + end +end diff --git a/test/integration/payload_delivery_test.rb b/test/integration/payload_delivery_test.rb index 5f9d182..e0937d7 100644 --- a/test/integration/payload_delivery_test.rb +++ b/test/integration/payload_delivery_test.rb @@ -3,6 +3,7 @@ require "database_test_helper" require "action_cable/test_helper" require "action_cable/channel/test_case" +require "portable_telemetry_assertions" ActionCable.server.config.cable = { "adapter" => "test" } @@ -11,6 +12,7 @@ # down with it. class PayloadDeliveryTest < ActionCable::Channel::TestCase tests SolidObjects::ActorChannel + include PortableTelemetryAssertions class RoomActor < SolidObjects::Actor actor_type "delivery-room" @@ -180,6 +182,19 @@ def initialize(session_id) assert_equal "RuntimeError", event[:error_class] end + test "a raising payload block emits the portable payload failure contract" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + reference = deal_to("alice") + stub_connection(session_id: "alice") + + subscribe token: payload_token(reference, %w[broken_state]) + + assert_portable_events(events, %w[payload_broadcast.failed]) + failure = events.find { |event| event.fetch("name") == "solid_objects.payload_broadcast.failed" } + assert_equal "broken_state", failure.fetch("attributes").fetch("payload") + end + test "the failure event carries no payload state" do reference = deal_to("alice", %w[secret-hand]) events = capture_failures do @@ -366,7 +381,7 @@ def component_token(reference, revision:) def capture_failures events = [] subscription = ActiveSupport::Notifications.subscribe( - "solid_objects.payload_broadcast_failed" + "solid_objects.payload_broadcast.failed" ) { |event| events << event.payload } yield events diff --git a/test/integration/process_lifecycle_test.rb b/test/integration/process_lifecycle_test.rb index a40fa6e..8a6cd49 100644 --- a/test/integration/process_lifecycle_test.rb +++ b/test/integration/process_lifecycle_test.rb @@ -1,8 +1,11 @@ # frozen_string_literal: true require "database_test_helper" +require "portable_telemetry_assertions" class ProcessLifecycleTest < ActiveSupport::TestCase + include PortableTelemetryAssertions + class RecoveryActor < SolidObjects::Actor actor_type "process-recovery" @@ -44,6 +47,7 @@ def run assert_equal({ "runs" => 1 }, instance.reload.state) assert events.any? { |event| event.fetch("name") == "solid_objects.recovery.reclaimed" && event.fetch("attempt") == 2 } + assert_portable_events(events, %w[recovery.reclaimed]) assert_equal 2, message.reload.attempt_count assert message.completed? ensure diff --git a/test/integration/synchronous_invocation_test.rb b/test/integration/synchronous_invocation_test.rb index f0f231b..e9d34cc 100644 --- a/test/integration/synchronous_invocation_test.rb +++ b/test/integration/synchronous_invocation_test.rb @@ -4,8 +4,11 @@ require "solid_objects/mailbox" require "solid_objects/synchronous_invocation" require "timeout" +require "portable_telemetry_assertions" class SynchronousInvocationTest < ActiveSupport::TestCase + include PortableTelemetryAssertions + class CounterActor < SolidObjects::Actor actor_type "synchronous-counter" @@ -451,6 +454,8 @@ def wait(timeout:) end test "sync bounds database lock waits while durably enqueueing" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } reference = CounterActor.ref("enqueue-locked") reference.increment instance = actor_instance("enqueue-locked") @@ -478,6 +483,9 @@ def wait(timeout:) assert_equal "synchronous-counter", error.actor_type assert_equal "enqueue-locked", error.actor_id assert_equal "increment", error.operation + assert_portable_events(events, %w[sync.enqueue_timeout]) + timeout = events.find { |event| event.fetch("name") == "solid_objects.sync.enqueue_timeout" } + assert_equal 250, timeout.fetch("attributes").fetch("timeoutMilliseconds") release_lock.push(true) blocker.join release_lock = nil diff --git a/test/integration/telemetry_test.rb b/test/integration/telemetry_test.rb index ca7cf74..ddc6255 100644 --- a/test/integration/telemetry_test.rb +++ b/test/integration/telemetry_test.rb @@ -1,8 +1,39 @@ # frozen_string_literal: true require "database_test_helper" +require "portable_telemetry_assertions" class TelemetryTest < ActiveSupport::TestCase + include PortableTelemetryAssertions + + class RecordingLogger + attr_reader :errors + + def initialize + @errors = [] + end + + def error(entry) + errors << entry + end + + def info(_entry) + end + + def warn(_entry) + end + end + + class ActivationFailure < SolidObjects::Actor + actor_type "telemetry-activation-failure" + + on_activate { raise "private activation failure" } + + def run + nil + end + end + class Counter < SolidObjects::Actor actor_type "telemetry-counter" attribute :count, default: 0 @@ -17,6 +48,20 @@ def fail_operation raise "private failure" end + def refuse + reject :refused, "private rejection" + end + + def commit + commit_action :telemetry_action + nil + end + + def commit_badly + commit_action :telemetry_failure + nil + end + def increment self.count += 1 end @@ -28,6 +73,120 @@ def increment backoff = SolidObjects::PollingBackoff.new(minimum_interval: 0.1, maximum_interval: 1, on_change: ->(transition) { SolidObjects.instrument(:"polling.interval_changed", role: "actors", **transition) }) backoff.record_idle assert_equal({ "role" => "actors", "previousIntervalMilliseconds" => 100, "currentIntervalMilliseconds" => 200, "reason" => "idle" }, events.last.fetch("attributes")) + assert_equal %({"role":"actors","previousIntervalMilliseconds":100,"currentIntervalMilliseconds":200,"reason":"idle"}), JSON.generate(events.last.fetch("attributes")) + assert_portable_attributes(events.last) + end + + test "the portable attribute allowlist matches the shared contract" do + assert_equal TELEMETRY_CONTRACT.fetch("attributes").sort, SolidObjects::Telemetry::FIELDS.sort + end + + test "portable SQL lifecycle events match the shared attribute contract" do + events = [] + SolidObjects.configuration.instrumentation = ->(event) { events << event } + SolidObjects.configuration.max_attempts = 2 + SolidObjects.configuration.retry_delay = ->(_) { 0 } + SolidObjects.configuration.authorize_administration = ->(**) { true } + SolidObjects.register_effect(:telemetry_effect) { "delivered" } + SolidObjects.register_commit_action(:telemetry_action) { |_arguments, _context| nil } + SolidObjects.register_commit_action(:telemetry_failure) { |_arguments, _context| raise "private commit failure" } + reference = Counter.ref("contract") + reference.increment + reference.async.refuse + reference.async.fail_operation + reference.async.commit + reference.async.commit_badly + reference.async.arrange + SolidObjects::Mailbox.new.enqueue(reference:, operation: "increment", arguments: {}, delivery_mode: "internal", idempotency_key: "effect:contract:recovery") + reference.diagnostics(limit: 1) + worker = SolidObjects::Worker.new + worker.run_until_idle + ActivationFailure.ref("contract").async.run + assert_raises(RuntimeError) { worker.run_once } + worker.stop + effect_executor = SolidObjects::EffectExecutor.new + effect_executor.run_once + scheduler = SolidObjects::ReminderScheduler.new + scheduler.run_once + reference.snapshot + + assert_portable_events(events, %w[ + activation.started activation.completed activation.failed + message.enqueued message.started message.completed message.rejected message.failed message.retry dead_letter.created + commit_action.started commit_action.completed commit_action.failed + recovery.completed mailbox.depth outbox.age reminder.enqueued snapshot.read + ]) + failures = events.select { |event| event.fetch("name") == "solid_objects.message.failed" }.map { |event| event.fetch("attributes").slice("retryable", "outcome") } + assert_includes failures, { "retryable" => true, "outcome" => "retrying" } + assert_includes failures, { "retryable" => true, "outcome" => "dead" } + depth = events.find { |event| event.fetch("name") == "solid_objects.mailbox.depth" }.fetch("attributes") + assert depth.fetch("truncated") + assert_nil depth.fetch("depth") + refute_includes events.to_json, "private" + ensure + worker&.stop + effect_executor&.stop + scheduler&.stop + end + + test "a failing exporter is logged and cannot fail a turn" do + logger = RecordingLogger.new + SolidObjects.configuration.logger = logger + SolidObjects.configuration.instrumentation = ->(_) { raise "private exporter failure" } + + assert_equal 1, Counter.ref("logged-exporter").increment + + assert_includes logger.errors, { event: "solid_objects.instrumentation.failed", instrumentation_event: "solid_objects.message.completed", error_class: "RuntimeError" } + refute_includes logger.errors.to_s, "private" + end + + test "a failing observer is logged and cannot fail a turn" do + logger = RecordingLogger.new + SolidObjects.configuration.logger = logger + SolidObjects.configuration.authorize_administration = ->(**) { true } + reference = Counter.ref("logged-observer") + stop = reference.observe { raise "private observer failure" } + + assert_equal 1, reference.increment + + assert_includes logger.errors, { event: "solid_objects.instrumentation.failed", instrumentation_event: "solid_objects.message.completed", error_class: "RuntimeError" } + refute_includes logger.errors.to_s, "private" + ensure + stop&.call + end + + test "observers require a block" do + SolidObjects.configuration.authorize_administration = ->(**) { true } + reference = Counter.ref("blockless") + + assert_raises(ArgumentError) { reference.observe } + assert_raises(ArgumentError) { reference.on("message.completed") } + end + + test "a process accepts at most 1000 local observers" do + SolidObjects.configuration.authorize_administration = ->(**) { true } + reference = Counter.ref("observer-limit") + stops = Array.new(1000) { reference.observe { nil } } + + error = assert_raises(ArgumentError) { reference.observe { nil } } + + assert_equal "at most 1000 local observers may be registered", error.message + stops.pop.call + stops << reference.observe { nil } + ensure + stops&.each(&:call) + end + + test "reset removes registered observers" do + SolidObjects.configuration.authorize_administration = ->(**) { true } + events = [] + Counter.ref("reset-observer").observe { |event| events << event } + + SolidObjects.reset! + SolidObjects.configuration.authorize_message = ->(**) { true } + Counter.ref("reset-observer").increment + + assert_empty events end JSON.parse(File.read(File.expand_path("../../compatibility/sync-timeout.json", __dir__))).each do |fixture| @@ -151,7 +310,7 @@ def increment worker = SolidObjects::Worker.new worker.run_once assert_equal 1, reference.diagnostics.fetch("recoveryFailures").fetch("sampled") - assert events.any? { |event| event.fetch("name") == "solid_objects.recovery.failed" } + assert_portable_events(events, %w[recovery.failed]) ensure worker&.stop end @@ -191,6 +350,7 @@ def increment assert executor.run_once event = events.find { |entry| entry.fetch("name") == "solid_objects.outbox.age" && entry.fetch("attributes")["outboxKind"] == "broadcast" } assert event + assert_portable_attributes(event) assert_operator event.fetch("metrics").last.fetch("value"), :>=, 0 ensure executor&.stop diff --git a/test/portable_telemetry_assertions.rb b/test/portable_telemetry_assertions.rb new file mode 100644 index 0000000..2e7649b --- /dev/null +++ b/test/portable_telemetry_assertions.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +module PortableTelemetryAssertions + TELEMETRY_CONTRACT = JSON.parse(File.read(File.expand_path("../compatibility/telemetry-events.json", __dir__))).freeze + + def assert_portable_attributes(event) + name = event.fetch("name").delete_prefix("solid_objects.") + attributes = event.fetch("attributes") + contract = TELEMETRY_CONTRACT.fetch("events").find do |entry| + entry.fetch("name") == name && entry.fetch("match", {}).all? { |key, value| attributes[key] == value } + end + + assert contract, "#{name} has no portable attribute contract" + assert_equal contract.fetch("attributes").sort, attributes.keys.sort, "#{name} attributes" + end + + def assert_portable_events(events, names) + names.each do |name| + matching = events.select { |event| event.fetch("name") == "solid_objects.#{name}" } + + refute_empty matching, "expected a solid_objects.#{name} event" + matching.each { |event| assert_portable_attributes(event) } + end + end +end diff --git a/test/unit/serialization_test.rb b/test/unit/serialization_test.rb index 3594523..64f2576 100644 --- a/test/unit/serialization_test.rb +++ b/test/unit/serialization_test.rb @@ -12,6 +12,12 @@ class SerializationTest < ActiveSupport::TestCase end end + test "includes reserved keys in the encoded byte limit" do + assert_raises(SolidObjects::PayloadTooLarge) do + SolidObjects::Serialization.dump({ "__proto__" => "long payload" }, max_bytes: 2) + end + end + test "normalizes symbol keys and nested values" do value = SolidObjects::Serialization.dump({ product_id: "shirt", From da1f00e19b71e0fb7546ede72a6e6c1632ab3d8c Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Thu, 1 Oct 2026 08:00:37 -0700 Subject: [PATCH 08/11] docs: Record activation and error payload changes Two behavior changes in this branch had no release note. The activation.started event now fires before the activate hook, and activation.completed takes its earlier meaning. Active Support payloads and the worker error log no longer carry error_message, because exception text can contain actor state. The shared observability guide had no Ruby setup sample and named UnsupportedCapability as if Ruby had it. Add the sample and mark the Durable Objects behavior as JavaScript only. --- CHANGELOG.md | 14 ++++++++++++++ docs/observability.md | 25 +++++++++++++++++++++---- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7317419..8fd4c5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ ## Unreleased +- `solid_objects.activation.started` now fires before the actor's `activate` + hook. Before, it fired after a successful hook. The new + `solid_objects.activation.completed` event takes that meaning, and + `solid_objects.activation.failed` reports a failed hook. JavaScript changes + the same events. Move a subscriber that reads `activation.started` as a + finished activation to `activation.completed`. +- Active Support payloads no longer carry `error_message`. This applies to + `commit_action.failed`, `activation.deactivation_failed`, + `supervisor.monitor_failed`, `supervisor.retention_failed`, + `supervisor.redrive_failed`, and `wake_up.failed`. The + `solid_objects.worker.error` log entry also omits it. Each keeps + `error_class`. Exception text can contain actor state, so JavaScript already + reports only the error name. + - Rename `solid_objects.payload_broadcast_failed` to `solid_objects.payload_broadcast.failed`, the dotted form that every other event uses. Update Active Support subscribers to the new name. The portable diff --git a/docs/observability.md b/docs/observability.md index feae949..af1c63c 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -6,6 +6,17 @@ and the Durable Objects host. Existing JavaScript `name`, `occurredAt`, and `attributes` fields remain available. Ruby's Active Support notifications remain available with their existing snake_case payloads. +```ruby +SolidObjects.configure do |configuration| + configuration.instrumentation = ->(event) do + Rails.logger.info(JSON.generate(event)) + end +end +``` + +JavaScript passes `instrumentation` to `configure()`. Ruby sets +`configuration.instrumentation` inside `SolidObjects.configure`. + ## Schema version 1 Every event has `schemaVersion`, `name` (prefixed with `solid_objects.`), @@ -118,10 +129,16 @@ calling the returned proc. `on` filters one event name, such as `message.retry`. Observers receive only this actor's events in the current runtime/process; they are not subscriptions to workers on other hosts. Dispose them when the caller's session ends or authorization is revoked. Each runtime or process accepts at most -1,000 local observers. An observer needs a callback or a block. -For remote Durable Objects, configure `instrumentation` on the actor host and -filter by actor identity there; process-local reference observers raise -`UnsupportedCapability`. Remote `reference.diagnostics` is supported. +1,000 local observers. More observers raise `RangeError` in JavaScript and +`ArgumentError` in Ruby. An observer needs a callback or a block. JavaScript +rejects a missing `onEvent` with `TypeError`, and Ruby raises `ArgumentError` +without a block. Both checks run before authorization. + +The JavaScript Durable Objects host does not support process-local reference +observers. On that host, `observe` and `on` raise `UnsupportedCapability`, and +remote `reference.diagnostics` works. To observe a remote actor, configure +`instrumentation` on the actor host and filter the events by actor identity. +Ruby has no Durable Objects host. Both APIs default to denied. Set `authorizeAdministration` / `authorize_administration` to allow action `observe` or `inspect`, resource `actor_diagnostics`, and resource ID From f31f50e42876a5d114ba365826d17ff97ec63007 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Thu, 1 Oct 2026 08:02:20 -0700 Subject: [PATCH 09/11] test: Share the packaged type check helpers The telemetry type contract needs the same packaged-gem Steep check as the effect payload contract. Move the helpers into one module so both tests use them. --- test/integration/effect_payload_types_test.rb | 50 ++---------------- test/packaged_type_check.rb | 51 +++++++++++++++++++ 2 files changed, 54 insertions(+), 47 deletions(-) create mode 100644 test/packaged_type_check.rb diff --git a/test/integration/effect_payload_types_test.rb b/test/integration/effect_payload_types_test.rb index c15fd2b..77884ec 100644 --- a/test/integration/effect_payload_types_test.rb +++ b/test/integration/effect_payload_types_test.rb @@ -1,13 +1,12 @@ # rbs_inline: enabled require "test_helper" -require "fileutils" -require "open3" require "tmpdir" -require "rubygems/package" -require "rubygems/installer" +require "packaged_type_check" class EffectPayloadTypesTest < ActiveSupport::TestCase + include PackagedTypeCheck + test "packaged payload contracts check consumers and the actual constructors" do Dir.mktmpdir("solid-objects-effect-types") do |directory| package = build_package(directory) @@ -74,47 +73,4 @@ class EffectPayloadTypesTest < ActiveSupport::TestCase end end end - - private - - def build_package(directory) - artifact = File.join(directory, "solid_objects.gem") - specification = Gem::Specification.load(root_path("solid_objects.gemspec")) - Gem::DefaultUserInteraction.use_ui(Gem::SilentUI.new) do - Gem::Package.build(specification, false, false, artifact) - end - artifact - end - - def typecheck(project) - output, error_output, status = Open3.capture3( - Gem.ruby, Gem.bin_path("steep", "steep"), "check", "--no-daemon", "-j", "1", - chdir: project - ) - [ output + error_output, status ] - end - - def typecheck_installed(project, package) - gem_directory = File.join(project, "gems") - specification = Gem::Installer.at(package, install_dir: gem_directory, ignore_dependencies: true).install - script = <<~RUBY - gem "solid_objects", #{"= #{SolidObjects::VERSION}".inspect} - resolved = Gem.loaded_specs.fetch("solid_objects").full_gem_path - abort "loaded signatures outside the built gem: \#{resolved}" unless resolved == #{specification.full_gem_path.inspect} - load ARGV.shift - RUBY - output, error_output, status = Open3.capture3( - { - "RUBYOPT" => nil, "BUNDLE_GEMFILE" => nil, - "GEM_HOME" => gem_directory, "GEM_PATH" => ([ gem_directory ] + Gem.path).join(File::PATH_SEPARATOR) - }, - Gem.ruby, "-e", script, Gem.bin_path("steep", "steep"), "check", "--no-daemon", "-j", "1", - chdir: project - ) - [ output + error_output, status ] - end - - def root_path(path) - File.expand_path("../../#{path}", __dir__) - end end diff --git a/test/packaged_type_check.rb b/test/packaged_type_check.rb new file mode 100644 index 0000000..ba9736d --- /dev/null +++ b/test/packaged_type_check.rb @@ -0,0 +1,51 @@ +# rbs_inline: enabled + +require "fileutils" +require "open3" +require "rubygems/package" +require "rubygems/installer" + +module PackagedTypeCheck + private + + def build_package(directory) + artifact = File.join(directory, "solid_objects.gem") + specification = Gem::Specification.load(root_path("solid_objects.gemspec")) + Gem::DefaultUserInteraction.use_ui(Gem::SilentUI.new) do + Gem::Package.build(specification, false, false, artifact) + end + artifact + end + + def typecheck(project) + output, error_output, status = Open3.capture3( + Gem.ruby, Gem.bin_path("steep", "steep"), "check", "--no-daemon", "-j", "1", + chdir: project + ) + [ output + error_output, status ] + end + + def typecheck_installed(project, package) + gem_directory = File.join(project, "gems") + specification = Gem::Installer.at(package, install_dir: gem_directory, ignore_dependencies: true).install + script = <<~RUBY + gem "solid_objects", #{"= #{SolidObjects::VERSION}".inspect} + resolved = Gem.loaded_specs.fetch("solid_objects").full_gem_path + abort "loaded signatures outside the built gem: \#{resolved}" unless resolved == #{specification.full_gem_path.inspect} + load ARGV.shift + RUBY + output, error_output, status = Open3.capture3( + { + "RUBYOPT" => nil, "BUNDLE_GEMFILE" => nil, + "GEM_HOME" => gem_directory, "GEM_PATH" => ([ gem_directory ] + Gem.path).join(File::PATH_SEPARATOR) + }, + Gem.ruby, "-e", script, Gem.bin_path("steep", "steep"), "check", "--no-daemon", "-j", "1", + chdir: project + ) + [ output + error_output, status ] + end + + def root_path(path) + File.expand_path("../#{path}", __dir__) + end +end From e67524818fa2095d5da6010add4b79c1ffa78214 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Thu, 1 Oct 2026 08:09:56 -0700 Subject: [PATCH 10/11] fix: Type the portable telemetry contracts Observer blocks, diagnostics, telemetry helpers, and message results used untyped, so Steep accepted a consumer that read the wrong field. sig/public/telemetry.rbs now publishes portable_event, metric_sample, event_observer, diagnostic_summary, and actor_diagnostics. A json_value type covers message results and actor state. A strict packaged type test checks a consumer and five invalid versions of it. With the old untyped observer block, the invalid observer consumer type-checked. authorization_context stays untyped, as on main, because it holds the application's own subject. --- CHANGELOG.md | 4 ++ app/models/solid_objects/message.rb | 2 +- docs/observability.md | 3 ++ lib/solid_objects/actor.rb | 4 +- lib/solid_objects/actor_snapshot.rb | 4 +- lib/solid_objects/configuration.rb | 5 +- lib/solid_objects/diagnostics.rb | 8 +-- lib/solid_objects/executor.rb | 2 +- lib/solid_objects/message_reference.rb | 2 +- lib/solid_objects/payload_broadcast.rb | 2 +- lib/solid_objects/reference.rb | 6 +-- lib/solid_objects/telemetry.rb | 12 ++--- sig/generated/lib/solid_objects/actor.rbs | 8 +-- .../lib/solid_objects/actor_snapshot.rbs | 6 +-- .../lib/solid_objects/configuration.rbs | 14 +++--- .../lib/solid_objects/diagnostics.rbs | 14 +++--- sig/generated/lib/solid_objects/executor.rbs | 4 +- .../lib/solid_objects/message_reference.rbs | 4 +- .../lib/solid_objects/payload_broadcast.rbs | 4 +- sig/generated/lib/solid_objects/reference.rbs | 12 ++--- sig/generated/lib/solid_objects/telemetry.rbs | 20 ++++---- .../models/solid_objects/message.rbs | 4 +- sig/public/json_value.rbs | 3 ++ sig/public/telemetry.rbs | 49 +++++++++++++++++++ test/integration/telemetry_types_test.rb | 43 ++++++++++++++++ test/types/telemetry.rb | 25 ++++++++++ test/types/telemetry.rbs | 7 +++ 27 files changed, 203 insertions(+), 68 deletions(-) create mode 100644 sig/public/json_value.rbs create mode 100644 sig/public/telemetry.rbs create mode 100644 test/integration/telemetry_types_test.rb create mode 100644 test/types/telemetry.rb create mode 100644 test/types/telemetry.rbs diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fd4c5b..f346015 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +- Publish RBS types for portable events, metric samples, actor diagnostics, and + event observers in `sig/public/telemetry.rbs`, and a `json_value` type for + message results and actor state. Observer blocks, diagnostics, and results now + type-check against these contracts instead of `untyped`. - `solid_objects.activation.started` now fires before the actor's `activate` hook. Before, it fired after a successful hook. The new `solid_objects.activation.completed` event takes that meaning, and diff --git a/app/models/solid_objects/message.rb b/app/models/solid_objects/message.rb index 5ab5a32..84351d7 100644 --- a/app/models/solid_objects/message.rb +++ b/app/models/solid_objects/message.rb @@ -47,7 +47,7 @@ def dead? dead_letter.present? end - # @rbs () -> untyped + # @rbs () -> json_value def result! if rejected? raise Rejected.new( diff --git a/docs/observability.md b/docs/observability.md index af1c63c..3892f2d 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -23,6 +23,9 @@ Every event has `schemaVersion`, `name` (prefixed with `solid_objects.`), `occurredAt` (UTC ISO 8601), `adapter`, `actorType`, `actorId`, `incarnation`, `revision`, `messageId`, `attempt`, `attributes`, and `metrics`. Unavailable identifiers are null; `attempt` is zero outside a message attempt. +Ruby publishes the RBS types `SolidObjects::portable_event`, +`SolidObjects::actor_diagnostics`, and `SolidObjects::event_observer`. JavaScript +exports `InstrumentationEvent`, `ActorDiagnostics`, and `EventObserver`. An incarnation identifies a persisted actor instance, independently of its lease generation. Revisions and IDs are strings. Process-wide events have null actor identity. A message ID or revision correlates actor work where applicable. diff --git a/lib/solid_objects/actor.rb b/lib/solid_objects/actor.rb index ffc5336..d921aaa 100644 --- a/lib/solid_objects/actor.rb +++ b/lib/solid_objects/actor.rb @@ -545,7 +545,7 @@ def intent_count reminder_intents.length + outbound_message_intents.length end - # @rbs (String) { () -> untyped } -> untyped + # @rbs [Result] (String) { () -> Result } -> Result def read_projection(operation) state_before = state.to_h intents_before = intent_snapshot @@ -565,7 +565,7 @@ def read_projection(operation) :reminder_intents, :outbound_message_intents - # @rbs () -> Array[Array[Hash[Symbol, untyped]]] + # @rbs () -> Array[Array[Hash[Symbol, Object]]] def intent_snapshot [ effect_intents, effect_recovery_intents, commit_action_intents, reminder_intents, outbound_message_intents ].map do |intents| intents.map { |intent| intent.to_h.deep_dup } diff --git a/lib/solid_objects/actor_snapshot.rb b/lib/solid_objects/actor_snapshot.rb index c2100a1..36727c2 100644 --- a/lib/solid_objects/actor_snapshot.rb +++ b/lib/solid_objects/actor_snapshot.rb @@ -7,7 +7,7 @@ class ActorSnapshot # @rbs @actor: Actor # @rbs @instance_id: Integer # @rbs @revision: Integer - # @rbs @state_data: Hash[String, untyped] + # @rbs @state_data: Hash[String, json_value] # @rbs @observable_values: Hash[String, untyped]? # @rbs @observable_value_cache: Hash[String, untyped] @@ -58,7 +58,7 @@ def build_actor attr_reader :instance - # @rbs () -> Hash[String, untyped] + # @rbs () -> Hash[String, json_value] def migrated_state state_version = instance&.state_version || actor_class.state_version ApplicationWriteGuard.call( diff --git a/lib/solid_objects/configuration.rb b/lib/solid_objects/configuration.rb index 6f320ae..0bfc1ba 100644 --- a/lib/solid_objects/configuration.rb +++ b/lib/solid_objects/configuration.rb @@ -40,7 +40,7 @@ class Configuration # @rbs @broadcast_worker_count: Integer # @rbs @reminder_scheduler_count: Integer # @rbs @connects_to: Hash[Symbol, untyped]? - # @rbs @instrumentation: Proc? + # @rbs @instrumentation: event_observer? # @rbs @logger: untyped # @rbs @stream_signing_secret: String? # @rbs @broadcast_adapter: Proc? @@ -96,7 +96,6 @@ class Configuration :reminder_scheduler_count, :connects_to, :logger, - :instrumentation, :stream_signing_secret, :broadcast_adapter, :wake_up_adapter, @@ -112,6 +111,8 @@ class Configuration :administration_identity, :transmission_actor_type_resolver + attr_accessor :instrumentation #: event_observer? + # @rbs @additional_components: Array[untyped] attr_reader :additional_components diff --git a/lib/solid_objects/diagnostics.rb b/lib/solid_objects/diagnostics.rb index 71d9c6c..cb09371 100644 --- a/lib/solid_objects/diagnostics.rb +++ b/lib/solid_objects/diagnostics.rb @@ -8,7 +8,7 @@ def initialize(reference) @reference = reference end - # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + # @rbs (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics def summary(limit: 100, authorization_context: nil) authorize!(:inspect, authorization_context:) raise ArgumentError, "diagnostic limit must be an integer between 1 and 100" unless limit.is_a?(Integer) && limit.between?(1, 100) @@ -42,7 +42,7 @@ def summary(limit: 100, authorization_context: nil) Serialization.readonly_copy(result) end - # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (?authorization_context: untyped) { (portable_event) -> void } -> Proc def observe(authorization_context: nil, &block) raise ArgumentError, "an actor observer requires a block" unless block @@ -57,7 +57,7 @@ def observe(authorization_context: nil, &block) private - attr_reader :reference + attr_reader :reference #: Reference # @rbs (Symbol, ?authorization_context: untyped) -> void def authorize!(action, authorization_context: nil) @@ -70,7 +70,7 @@ def authorize!(action, authorization_context: nil) raise Unauthorized, "actor diagnostics are not authorized" unless allowed end - # @rbs (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] + # @rbs (Array[Time?], now: Time, limit: Integer) -> diagnostic_summary def summarize(timestamps, now:, limit:) oldest = timestamps.compact.min { diff --git a/lib/solid_objects/executor.rb b/lib/solid_objects/executor.rb index 10adce1..b93d2d7 100644 --- a/lib/solid_objects/executor.rb +++ b/lib/solid_objects/executor.rb @@ -69,7 +69,7 @@ def invoke_actor(message_context) end end - # @rbs (Hash[String, untyped], Hash[String, untyped]) -> void + # @rbs (Hash[String, json_value], Hash[String, json_value]) -> void def ensure_query_is_read_only!(state_before, state_after) return unless actor.class.definition.queries.key?(message.operation.to_sym) diff --git a/lib/solid_objects/message_reference.rb b/lib/solid_objects/message_reference.rb index ade6048..b77a14a 100644 --- a/lib/solid_objects/message_reference.rb +++ b/lib/solid_objects/message_reference.rb @@ -38,7 +38,7 @@ def status(authorization_context: nil) status_of(SolidObjects.client.read_message(self, authorization_context:)) end - # @rbs (?authorization_context: untyped) -> untyped + # @rbs (?authorization_context: untyped) -> json_value def result(authorization_context: nil) SolidObjects.client.read_message(self, authorization_context:).result! end diff --git a/lib/solid_objects/payload_broadcast.rb b/lib/solid_objects/payload_broadcast.rb index d441d0b..3b59097 100644 --- a/lib/solid_objects/payload_broadcast.rb +++ b/lib/solid_objects/payload_broadcast.rb @@ -71,7 +71,7 @@ def evaluated_payload(handler) # Distinguishes a block that relied on the old class-level receiver from an # ordinary typo, so the one behaviour change reports itself instead of # surfacing as an unexplained NameError. - # @rbs (NameError[untyped], Actor?) -> bool + # @rbs (NameError[BasicObject], Actor?) -> bool def class_level_receiver?(error, actor) error.receiver.equal?(actor) && snapshot.actor_class.respond_to?(error.name) diff --git a/lib/solid_objects/reference.rb b/lib/solid_objects/reference.rb index e92bd28..91faec2 100644 --- a/lib/solid_objects/reference.rb +++ b/lib/solid_objects/reference.rb @@ -16,17 +16,17 @@ def initialize(actor_type:, actor_id:) freeze end - # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + # @rbs (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics def diagnostics(limit: 100, authorization_context: nil) Diagnostics.new(self).summary(limit:, authorization_context:) end - # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (?authorization_context: untyped) { (portable_event) -> void } -> Proc def observe(authorization_context: nil, &block) Diagnostics.new(self).observe(authorization_context:, &block) end - # @rbs (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (String, ?authorization_context: untyped) { (portable_event) -> void } -> Proc def on(name, authorization_context: nil, &block) raise ArgumentError, "an actor observer requires a block" unless block diff --git a/lib/solid_objects/telemetry.rb b/lib/solid_objects/telemetry.rb index 7e2a140..e761061 100644 --- a/lib/solid_objects/telemetry.rb +++ b/lib/solid_objects/telemetry.rb @@ -11,14 +11,14 @@ module Telemetry phase pollingIntervalMilliseconds previousIntervalMilliseconds previousRunAt processId processKind reason reminderId reminderSchedulers requestId retryable revision role sequence status thresholdBytes timeoutMilliseconds truncated waitingOn workers - ].freeze + ].freeze #: Array[String] ALIASES = { "errorClass" => "errorName", "stateRevision" => "revision", "durationMs" => "durationMilliseconds", "commitActionName" => "commitAction", "payloadName" => "payload" - }.freeze + }.freeze #: Hash[String, String] class << self # @rbs (Effect | Broadcast) -> void @@ -41,7 +41,7 @@ def outbox(record) nil end - # @rbs (Symbol, Hash[Symbol, untyped]) -> void + # @rbs (Symbol, Hash[Symbol, Object]) -> void def emit(name, payload) observer = SolidObjects.configuration.instrumentation return unless observer @@ -51,14 +51,14 @@ def emit(name, payload) nil end - # @rbs (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void + # @rbs (event_observer, portable_event) -> void def deliver(observer, event) observer.call(event) rescue => error log_delivery_failure(event.fetch("name"), error) end - # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] + # @rbs (Symbol, Hash[Symbol, Object]) -> portable_event def event(name, payload) attributes = safe_attributes(payload) adapter = DatabaseAdapter.family(Record.connection).to_s @@ -92,7 +92,7 @@ def event(name, payload) ) end - # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] + # @rbs (Hash[Symbol, Object]) -> Hash[String, telemetry_value] def safe_attributes(payload) payload.each_with_object({}) do |(key, value), attributes| value = value.to_s if key == :reason && value.is_a?(Symbol) diff --git a/sig/generated/lib/solid_objects/actor.rbs b/sig/generated/lib/solid_objects/actor.rbs index 6915848..ed9b5cf 100644 --- a/sig/generated/lib/solid_objects/actor.rbs +++ b/sig/generated/lib/solid_objects/actor.rbs @@ -351,8 +351,8 @@ module SolidObjects # @rbs () -> Integer def intent_count: () -> Integer - # @rbs (String) { () -> untyped } -> untyped - def read_projection: (String) { () -> untyped } -> untyped + # @rbs [Result] (String) { () -> Result } -> Result + def read_projection: [Result] (String) { () -> Result } -> Result private @@ -366,8 +366,8 @@ module SolidObjects attr_reader outbound_message_intents: untyped - # @rbs () -> Array[Array[Hash[Symbol, untyped]]] - def intent_snapshot: () -> Array[Array[Hash[Symbol, untyped]]] + # @rbs () -> Array[Array[Hash[Symbol, Object]]] + def intent_snapshot: () -> Array[Array[Hash[Symbol, Object]]] # @rbs (String) { () -> untyped } -> untyped def guard_application_writes: (String) { () -> untyped } -> untyped diff --git a/sig/generated/lib/solid_objects/actor_snapshot.rbs b/sig/generated/lib/solid_objects/actor_snapshot.rbs index d0de883..0b07c23 100644 --- a/sig/generated/lib/solid_objects/actor_snapshot.rbs +++ b/sig/generated/lib/solid_objects/actor_snapshot.rbs @@ -12,7 +12,7 @@ module SolidObjects @revision: Integer - @state_data: Hash[String, untyped] + @state_data: Hash[String, json_value] @observable_values: Hash[String, untyped]? @@ -44,7 +44,7 @@ module SolidObjects attr_reader instance: untyped - # @rbs () -> Hash[String, untyped] - def migrated_state: () -> Hash[String, untyped] + # @rbs () -> Hash[String, json_value] + def migrated_state: () -> Hash[String, json_value] end end diff --git a/sig/generated/lib/solid_objects/configuration.rbs b/sig/generated/lib/solid_objects/configuration.rbs index fb9989e..a709bb4 100644 --- a/sig/generated/lib/solid_objects/configuration.rbs +++ b/sig/generated/lib/solid_objects/configuration.rbs @@ -2,8 +2,6 @@ module SolidObjects class Configuration - @transmission_actor_type_resolver: Proc - @reminder_scheduler_count: Integer @broadcast_worker_count: Integer @@ -36,6 +34,10 @@ module SolidObjects @supervisor_monitor_interval: Float + @table_name_prefix: String + + @transmission_actor_type_resolver: Proc + @administration_identity: Proc @authorize_transmission: Proc @@ -64,12 +66,10 @@ module SolidObjects @logger: untyped - @instrumentation: Proc? + @instrumentation: event_observer? @connects_to: Hash[Symbol, untyped]? - @table_name_prefix: String - @polling_interval: Float @idle_polling_interval: Float @@ -188,8 +188,6 @@ module SolidObjects attr_accessor logger: untyped - attr_accessor instrumentation: untyped - attr_accessor stream_signing_secret: untyped attr_accessor broadcast_adapter: untyped @@ -218,6 +216,8 @@ module SolidObjects attr_accessor transmission_actor_type_resolver: untyped + attr_accessor instrumentation: event_observer? + # @rbs @additional_components: Array[untyped] attr_reader additional_components: untyped diff --git a/sig/generated/lib/solid_objects/diagnostics.rbs b/sig/generated/lib/solid_objects/diagnostics.rbs index fea1ec8..a554a8f 100644 --- a/sig/generated/lib/solid_objects/diagnostics.rbs +++ b/sig/generated/lib/solid_objects/diagnostics.rbs @@ -6,20 +6,20 @@ module SolidObjects # @rbs (Reference) -> void def initialize: (Reference) -> void - # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] - def summary: (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + # @rbs (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics + def summary: (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics - # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc - def observe: (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (?authorization_context: untyped) { (portable_event) -> void } -> Proc + def observe: (?authorization_context: untyped) { (portable_event) -> void } -> Proc private - attr_reader reference: untyped + attr_reader reference: Reference # @rbs (Symbol, ?authorization_context: untyped) -> void def authorize!: (Symbol, ?authorization_context: untyped) -> void - # @rbs (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] - def summarize: (Array[Time?], now: Time, limit: Integer) -> Hash[String, untyped] + # @rbs (Array[Time?], now: Time, limit: Integer) -> diagnostic_summary + def summarize: (Array[Time?], now: Time, limit: Integer) -> diagnostic_summary end end diff --git a/sig/generated/lib/solid_objects/executor.rbs b/sig/generated/lib/solid_objects/executor.rbs index 22d68ae..f4250b9 100644 --- a/sig/generated/lib/solid_objects/executor.rbs +++ b/sig/generated/lib/solid_objects/executor.rbs @@ -28,8 +28,8 @@ module SolidObjects # @rbs (MessageContext) -> untyped def invoke_actor: (MessageContext) -> untyped - # @rbs (Hash[String, untyped], Hash[String, untyped]) -> void - def ensure_query_is_read_only!: (Hash[String, untyped], Hash[String, untyped]) -> void + # @rbs (Hash[String, json_value], Hash[String, json_value]) -> void + def ensure_query_is_read_only!: (Hash[String, json_value], Hash[String, json_value]) -> void # @rbs (Hash[String, untyped], Hash[String, untyped]) -> Hash[String, untyped] def changed_observables: (Hash[String, untyped], Hash[String, untyped]) -> Hash[String, untyped] diff --git a/sig/generated/lib/solid_objects/message_reference.rbs b/sig/generated/lib/solid_objects/message_reference.rbs index 097bde6..e99a65b 100644 --- a/sig/generated/lib/solid_objects/message_reference.rbs +++ b/sig/generated/lib/solid_objects/message_reference.rbs @@ -31,8 +31,8 @@ module SolidObjects # @rbs (?authorization_context: untyped) -> String def status: (?authorization_context: untyped) -> String - # @rbs (?authorization_context: untyped) -> untyped - def result: (?authorization_context: untyped) -> untyped + # @rbs (?authorization_context: untyped) -> json_value + def result: (?authorization_context: untyped) -> json_value # @rbs (?authorization_context: untyped) -> Outcome def outcome: (?authorization_context: untyped) -> Outcome diff --git a/sig/generated/lib/solid_objects/payload_broadcast.rbs b/sig/generated/lib/solid_objects/payload_broadcast.rbs index 041daa4..e43d8d8 100644 --- a/sig/generated/lib/solid_objects/payload_broadcast.rbs +++ b/sig/generated/lib/solid_objects/payload_broadcast.rbs @@ -37,8 +37,8 @@ module SolidObjects # Distinguishes a block that relied on the old class-level receiver from an # ordinary typo, so the one behaviour change reports itself instead of # surfacing as an unexplained NameError. - # @rbs (NameError[untyped], Actor?) -> bool - def class_level_receiver?: (NameError[untyped], Actor?) -> bool + # @rbs (NameError[BasicObject], Actor?) -> bool + def class_level_receiver?: (NameError[BasicObject], Actor?) -> bool # @rbs () -> void def authorize!: () -> void diff --git a/sig/generated/lib/solid_objects/reference.rbs b/sig/generated/lib/solid_objects/reference.rbs index 834ea37..00230fd 100644 --- a/sig/generated/lib/solid_objects/reference.rbs +++ b/sig/generated/lib/solid_objects/reference.rbs @@ -13,14 +13,14 @@ module SolidObjects # @rbs (actor_type: String, actor_id: String) -> void def initialize: (actor_type: String, actor_id: String) -> void - # @rbs (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] - def diagnostics: (?limit: Integer, ?authorization_context: untyped) -> Hash[String, untyped] + # @rbs (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics + def diagnostics: (?limit: Integer, ?authorization_context: untyped) -> actor_diagnostics - # @rbs (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc - def observe: (?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (?authorization_context: untyped) { (portable_event) -> void } -> Proc + def observe: (?authorization_context: untyped) { (portable_event) -> void } -> Proc - # @rbs (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc - def on: (String, ?authorization_context: untyped) { (Hash[String, untyped]) -> untyped } -> Proc + # @rbs (String, ?authorization_context: untyped) { (portable_event) -> void } -> Proc + def on: (String, ?authorization_context: untyped) { (portable_event) -> void } -> Proc # @rbs (?available_at: Time?, ?idempotency_key: String?, ?authorization_context: untyped) -> OperationDispatcher def async: (?available_at: Time?, ?idempotency_key: String?, ?authorization_context: untyped) -> OperationDispatcher diff --git a/sig/generated/lib/solid_objects/telemetry.rbs b/sig/generated/lib/solid_objects/telemetry.rbs index f080d3c..97f2458 100644 --- a/sig/generated/lib/solid_objects/telemetry.rbs +++ b/sig/generated/lib/solid_objects/telemetry.rbs @@ -2,24 +2,24 @@ module SolidObjects module Telemetry - FIELDS: untyped + FIELDS: Array[String] - ALIASES: untyped + ALIASES: Hash[String, String] # @rbs (Effect | Broadcast) -> void def self.outbox: (Effect | Broadcast) -> void - # @rbs (Symbol, Hash[Symbol, untyped]) -> void - def self.emit: (Symbol, Hash[Symbol, untyped]) -> void + # @rbs (Symbol, Hash[Symbol, Object]) -> void + def self.emit: (Symbol, Hash[Symbol, Object]) -> void - # @rbs (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void - def self.deliver: (^(Hash[String, untyped]) -> void, Hash[String, untyped]) -> void + # @rbs (event_observer, portable_event) -> void + def self.deliver: (event_observer, portable_event) -> void - # @rbs (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] - def self.event: (Symbol, Hash[Symbol, untyped]) -> Hash[String, untyped] + # @rbs (Symbol, Hash[Symbol, Object]) -> portable_event + def self.event: (Symbol, Hash[Symbol, Object]) -> portable_event - # @rbs (Hash[Symbol, untyped]) -> Hash[String, untyped] - def self.safe_attributes: (Hash[Symbol, untyped]) -> Hash[String, untyped] + # @rbs (Hash[Symbol, Object]) -> Hash[String, telemetry_value] + def self.safe_attributes: (Hash[Symbol, Object]) -> Hash[String, telemetry_value] # @rbs (Effect | Broadcast) -> Hash[Symbol, String | Integer] private def self.outbox_identity: (Effect | Broadcast) -> Hash[Symbol, String | Integer] diff --git a/sig/generated/models/solid_objects/message.rbs b/sig/generated/models/solid_objects/message.rbs index 9d9e572..87b65f6 100644 --- a/sig/generated/models/solid_objects/message.rbs +++ b/sig/generated/models/solid_objects/message.rbs @@ -17,8 +17,8 @@ module SolidObjects # @rbs () -> bool def dead?: () -> bool - # @rbs () -> untyped - def result!: () -> untyped + # @rbs () -> json_value + def result!: () -> json_value private diff --git a/sig/public/json_value.rbs b/sig/public/json_value.rbs new file mode 100644 index 0000000..8829734 --- /dev/null +++ b/sig/public/json_value.rbs @@ -0,0 +1,3 @@ +module SolidObjects + type json_value = String | Integer | Float | bool | nil | Array[json_value] | Hash[String, json_value] +end diff --git a/sig/public/telemetry.rbs b/sig/public/telemetry.rbs new file mode 100644 index 0000000..13a6bf3 --- /dev/null +++ b/sig/public/telemetry.rbs @@ -0,0 +1,49 @@ +module SolidObjects + type telemetry_value = String | Integer | Float | bool | nil + + type metric_sample = { + "name" => String, + "kind" => "counter" | "gauge" | "histogram", + "unit" => "1" | "ms", + "value" => Integer | Float, + "labels" => Hash[String, String] + } + + type portable_event = { + "schemaVersion" => Integer, + "name" => String, + "occurredAt" => String, + "adapter" => String, + "actorType" => String?, + "actorId" => String?, + "incarnation" => String?, + "revision" => String?, + "messageId" => String?, + "attempt" => Integer, + "attributes" => Hash[String, telemetry_value], + "metrics" => Array[metric_sample] + } + + type event_observer = ^(portable_event) -> void + + type diagnostic_summary = { + "sampled" => Integer, + "truncated" => bool, + "oldestAgeMilliseconds" => Integer? + } + + type actor_diagnostics = { + "actorType" => String, + "actorId" => String, + "incarnation" => String?, + "revision" => String?, + "adapter" => String, + "occurredAt" => String, + "limit" => Integer, + "mailbox" => diagnostic_summary, + "outbox" => diagnostic_summary, + "reminders" => diagnostic_summary, + "retries" => diagnostic_summary, + "recoveryFailures" => diagnostic_summary + } +end diff --git a/test/integration/telemetry_types_test.rb b/test/integration/telemetry_types_test.rb new file mode 100644 index 0000000..0750b3b --- /dev/null +++ b/test/integration/telemetry_types_test.rb @@ -0,0 +1,43 @@ +# rbs_inline: enabled + +require "test_helper" +require "tmpdir" +require "packaged_type_check" + +class TelemetryTypesTest < ActiveSupport::TestCase + include PackagedTypeCheck + + test "packaged telemetry contracts check event and diagnostic consumers" do + Dir.mktmpdir("solid-objects-telemetry-types") do |directory| + project = File.join(directory, "consumer") + Gem::Package.new(build_package(directory)).extract_files(project) + FileUtils.cp(root_path("test/types/telemetry.rbs"), File.join(project, "sig/consumer.rbs")) + consumer_path = File.join(project, "consumer.rb") + consumer = File.read(root_path("test/types/telemetry.rb")) + File.write(consumer_path, consumer) + File.write(File.join(project, "Steepfile"), <<~RUBY) + target :consumer do + signature "sig" + check "consumer.rb" + configure_code_diagnostics(Diagnostic::Ruby.strict) + end + RUBY + + output, status = typecheck(project) + assert status.success?, output + + [ + [ 'event["name"]', 'event["attempt"]', "Ruby::MethodBodyTypeMismatch" ], + [ 'event["attributes"]["waitingOn"]', 'event["attributes"]', "Ruby::MethodBodyTypeMismatch" ], + [ 'metric["name"]', 'metric["value"]', "Ruby::BlockBodyTypeMismatch" ], + [ 'summary["mailbox"]["oldestAgeMilliseconds"]', 'summary["mailbox"]["truncated"]', "Ruby::MethodBodyTypeMismatch" ], + [ 'names << event["name"]', 'names << event["attempt"]', "Ruby::ArgumentTypeMismatch" ] + ].each do |original, invalid, diagnostic| + File.write(consumer_path, consumer.sub(original, invalid)) + output, status = typecheck(project) + refute status.success?, "invalid consumer escaped the compiler: #{invalid}" + assert_includes output, diagnostic + end + end + end +end diff --git a/test/types/telemetry.rb b/test/types/telemetry.rb new file mode 100644 index 0000000..3fbc89c --- /dev/null +++ b/test/types/telemetry.rb @@ -0,0 +1,25 @@ +# rbs_inline: enabled + +class TelemetryConsumer + def event_name(event) + event["name"] + end + + def wait_reason(event) + event["attributes"]["waitingOn"] + end + + def metric_names(event) + event["metrics"].map { |metric| metric["name"] } + end + + def oldest_mailbox_age(summary) + summary["mailbox"]["oldestAgeMilliseconds"] + end + + def observed_names(reference) + names = [] #: Array[String] + reference.observe(authorization_context: nil) { |event| names << event["name"] } + names + end +end diff --git a/test/types/telemetry.rbs b/test/types/telemetry.rbs new file mode 100644 index 0000000..fbf4732 --- /dev/null +++ b/test/types/telemetry.rbs @@ -0,0 +1,7 @@ +class TelemetryConsumer + def event_name: (SolidObjects::portable_event) -> String + def wait_reason: (SolidObjects::portable_event) -> SolidObjects::telemetry_value + def metric_names: (SolidObjects::portable_event) -> Array[String] + def oldest_mailbox_age: (SolidObjects::actor_diagnostics) -> Integer? + def observed_names: (SolidObjects::Reference) -> Array[String] +end From 08cd4274057712a2588d483c861c90aba0359cae Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Thu, 1 Oct 2026 08:35:25 -0700 Subject: [PATCH 11/11] test: Widen the sync lock deadline budget One 0.25 s deadline covered two steps: the actor start and the lock wait under test. When a slow CI runner took more than 0.25 s to start the actor, the call timed out before the actor ran, and the test waited 2 s for it at line 422. This failed the Rails 7.1 compatibility job in run 36877758705. Give the call 1 s, and set the outer limits to 3 s and 2 s. A 0.3 s stall before the actor claim reproduced the CI error with the old budget; with the new budget, 0.3 s and 0.6 s stalls pass. The limits stay below the 5 s SQLite busy timeout: when the deadline never expires, the test fails at line 443 on Rails 7.1 and 8.1. --- test/integration/synchronous_invocation_test.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/test/integration/synchronous_invocation_test.rb b/test/integration/synchronous_invocation_test.rb index e9d34cc..ced9fda 100644 --- a/test/integration/synchronous_invocation_test.rb +++ b/test/integration/synchronous_invocation_test.rb @@ -415,7 +415,7 @@ def wait(timeout:) result = Queue.new invocation = Thread.new do result << capture_exception do - SolidObjects::SynchronousInvocation.new.call(message_reference, timeout: 0.25) + SolidObjects::SynchronousInvocation.new.call(message_reference, timeout: 1) end end @@ -440,11 +440,11 @@ def wait(timeout:) started_at = monotonic_now DeadlineActor.continue << true - error = Timeout.timeout(2) { result.pop } + error = Timeout.timeout(3) { result.pop } elapsed = monotonic_now - started_at assert_instance_of SolidObjects::SyncTimeout, error - assert_operator elapsed, :<, 1.5 + assert_operator elapsed, :<, 2 assert_equal message_reference.id, error.message_id ensure DeadlineActor.continue << true if invocation&.alive?