fix(delayed_job): stop losing jobs to SIGTERM, report permanent failures

[mroderick]

Closes #2942

Removing Delayed::Worker.raise_signal_exceptions = :term changes what a SIGTERM does to the job pipeline. Reviewers should focus on that behaviour change, summarised here.

Behaviour change

Before: a TERM signal raised a SignalException inside the worker, interrupting the job in flight. The interrupted job consumed one of three attempts; after three such events the job was permanently marked failed and its side effect (typically an invitation email) was silently lost.

After: on TERM the worker finishes the current job, then exits. Any jobs still queued are picked up by the next Heroku Scheduler run (rake jobs:workoff, every 10 minutes), so a kill during an invitation send delays delivery by minutes instead of losing emails. Heroku's SIGKILL (30s after an unhandled TERM) is also recoverable: Delayed::Plugins::ClearLocks clears stale locks at worker boot.

Why the setting can be removed safely here

• The scheduler one-off dynos exit on their own when out of jobs, so nothing relies on TERM forcing a fast exit.
• Per-job work is short (render + SendGrid call, well inside the 30s TERM-to-SIGKILL window).
• The setting predates the scheduler architecture: it was added in April 2015 (commit dddc1a3e, PR Revamp #230) for a persistent worker-dyno setup, and PR Procfile: Drop the Worker process type, and use Heroku Scheduler instead #2294's move to scheduler draining changed its blast radius without anyone revisiting it.

Also in this PR

• A Delayed::Worker.lifecycle.after(:failure) hook reports permanently failed jobs (attempts exhausted) to Rollbar with the job id, attempt count, first line of the error, and first line of the handler, so silent accumulation like the current 2,184-row pile cannot happen unobserved.
• A delayed_jobs:prune_failed[before] rake task deletes failed jobs older than an optional cutoff (one year by default), for housekeeping of the existing pile.

Deliberately not done

• Requeueing any of the 2,184 failed jobs — they are old invitation/email jobs; requeueing needs its own triage (noted in Delayed jobs marked failed on SIGTERM — invitation emails silently lost during restarts #2942).
• Making parent send_*_emails jobs resumable — the September 2026 InvitationManager changes (commit 26d94993) already make re-runs safe.

Verification detail

The :failure callback signature (worker, job) and its post-fail! timing were verified against delayed_job 4.2.0 (lib/delayed/worker.rb#failed, lib/delayed/lifecycle.rb). In production the only TERM sources for job-draining dynos are the scheduler's 1-hour one-off cap and manual dyno stops; deploys do not restart scheduler dynos.

New specs cover: the removed setting, the Rollbar hook invocation, and the rake task (default cutoff, explicit cutoff, cutoff before all failures). Full suite: 1,557 examples, 0 failures; RuboCop clean on changed files.

[mroderick]

Follow-up: pruning the existing failed-jobs pile (for after this merges and deploys)

Preview the count before deleting anything:

heroku pg:psql --app codebar-production -c "SELECT count(*) FROM delayed_jobs WHERE failed_at IS NOT NULL AND failed_at < '2025-01-01';"

Then delete with the new rake task:

heroku run "bundle exec rake delayed_jobs:prune_failed[2025-01-01]" -a codebar-production

The 2025-01-01 cutoff keeps the recent SIGTERM failures referenced in #2942 (evidence for this PR) while dropping the bulk of the pile, most of which is from a single 2020 incident.

[olleolleolle]

Looks good!

[mroderick]

I ran the commands after deployment

heroku pg:psql --app codebar-production -c "SELECT count(*) FROM delayed_jobs WHERE failed_at IS NOT NULL AND failed_at < '2025-01-01';"
--> Connecting to ⛁ postgresql-infinite-55100
 count
-------
  2113
(1 row)

heroku run "bundle exec rake delayed_jobs:prune_failed[2025-01-01]" -a codebar-production
Running bundle exec rake delayed_jobs:prune_failed[2025-01-01] on ⬢ codebar-production... up, run.4857
deleted=2113 cutoff=2025-01-01

heroku pg:psql --app codebar-production -c "SELECT count(*) FROM delayed_jobs WHERE failed_at IS NOT NULL AND failed_at < '2025-01-01';"
--> Connecting to ⛁ postgresql-infinite-55100
 count
-------
     0
(1 row)