Background Jobs in Ruby on Rails

The low-level floor this page stands on

• Processes vs threadsmissing A process is a running program with its own private memory; if it dies, nothing else notices. A thread lives inside a process and shares its memory with every other thread there — cheaper to create, riskier to share. Puma and Sidekiq are thread-based; Unicorn and Resque are process-based. Every trade-off below reduces to this split.
• Blocking I/Omissing When your code queries Postgres, the CPU doesn't "work slower" — that thread does nothing at all until bytes come back. A 40 ms query is an eternity of idle CPU. Concurrency exists to spend those waits running someone else's code.
• The GIL (today: GVL)shaky MRI's Global VM Lock lets exactly one thread execute Ruby at any instant. It doesn't forbid threads; it schedules them. You could name it this morning but not explain it — §2 exists for you.
• Connection poolsuntested A pool is a box of N reusable connections; a thread checks one out, uses it, puts it back. When more threads want connections than the box holds, they queue — and after 5 seconds ActiveRecord raises the exact error you couldn't diagnose this morning. §4.

• 1What a job actually isstable
• 2The lie in concurrency: 25rewritten today
• 3Your InfoTech incident, decodedfrom memory
• 4Pools: the diagnostic you missednew today
• 5Sidekiq vs Solid Queue, 2026decision pending
• 6Open questions3 open

What a job actually is

Strip the mystique first. A background job is three ordinary things: a piece of JSON describing work — class name, arguments, queue —; a data structure holding it — a Redis list, or a Postgres table —; and a separate OS process that loops forever: pop, deserialize, call perform, repeat.

UserMailer.welcome(user).deliver_later does not send an email. It serializes intent and pushes it onto a list. Your web request answers fast because it did almost nothing. Everything else in this page is what happens inside that third thing — the process you never see.

The lie in concurrency: 25

This morning I asked why 25 threads make progress under a lock, and you chose "not clear to me." Fair. Here is the model, once, properly.

Under MRI, only one thread executes Ruby at any instant — that's the GVL. But the moment a thread performs I/O — a Postgres query, an HTTP call, a Redis command — it releases the lock and goes to sleep until the kernel has bytes for it. Another thread grabs the lock and runs. Sidekiq jobs are overwhelmingly I/O-bound, so 25 threads means 25 overlapping waits and roughly one thread executing Ruby at a time. That is concurrency during I/O waits — not CPU parallelism.

Your kitchen version: one chef (the interpreter), twenty-five pans (threads). The chef never fries two eggs simultaneously — but while pan 7 sizzles unattended (I/O), the chef stirs pan 12. A kitchen that is all chopping and no sizzling — CPU-bound work like image processing — gains nothing from more pans. It needs more chefs: processes, each with its own GVL. That is why Sidekiq lets you run several processes, and why Resque chose forking outright.

Your InfoTech incident, decoded

At InfoTech you ran a multitenant app where jobs kept dying, and you re-enqueued them by hand. You lived through three concepts that day without their names. Let's attach them.

Name one: at-least-once delivery. Sidekiq promises your job runs at least once — never exactly once. Crashes mid-run mean re-runs; re-runs mean duplicates must be survivable. Name two: the retry schedule. A failing job retries ~25 times over ~20 days with exponential backoff, then lands in the Dead set. Your manual re-enqueues were you re-implementing that machinery — without its bookkeeping. Name three: ambient context loss. Anything that lived in thread-local state — the current tenant, above all — is gone when the job wakes up elsewhere. Multitenant jobs that "die mysteriously" are usually jobs that woke up stateless.

The cure has a name too: idempotency. Design perform so that running twice equals running once — upserts instead of creates, guard clauses on state (return if invoice.sent?), uniqueness keys for the truly critical. Once a job is idempotent, retries stop being scary and become what they were meant to be: free reliability.

Pools: the diagnostic you missed

Every thread that touches ActiveRecord checks out a database connection and keeps it for the duration of its work. The pool size comes from database.yml — historically pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>. Raise Sidekiq to 30 threads without touching that number and you have built a bottleneck on purpose: 30 workers, 5 connections, 25 threads queuing at the box. Five seconds later, the exception.

The arithmetic that matters end-to-end: total Postgres connections ≈ (pool × Sidekiq processes) + (pool × web processes), and that sum must clear the server's max_connections — which is why teams put PgBouncer in front long before they "need microservices." Redis has a pool too, but Sidekiq sizes its own to match concurrency; the pool you personally own is ActiveRecord's.

The instinct you lacked this morning was not knowledge of an obscure flag. It was a habit: read the error class and ask who hands out the resource it names. ConnectionTimeoutError names a connection pool. Start there; compare concurrency vs pool; check the server's headroom; only then blame Redis, the GVL, or the infra. That four-step ritual is the troubleshooting axis on your radar.

Sidekiq vs Solid Queue, 2026

You know Solid Queue exists — Rails 8 made it the default — but not its internals. Here is the decision, compressed. Solid Queue keeps jobs in Postgres; workers claim them with SELECT … FOR UPDATE SKIP LOCKED, a row-level "take what nobody holds, skip what somebody does" that avoids both double-claims and lock convoys. No Redis, no extra process to babysit, and you can enqueue inside the same transaction as your domain data — the job cannot exist unless the order does. With Redis-backed Sidekiq, that atomicity is your discipline (after_commit), not the system's.

Open questions

• Q1Does Graphiti ingestion need a queue at all yet — or does your existing 7:30 cron pattern cover it until mysecondbrain has a second user?
• Q2What does GVL contention look like measured (Ruby 3.2+ GVL instrumentation, gvl_timing) on your actual workload, instead of assumed?
• Q3If mysecondbrain ever hosts other people's vaults: does the tenant key travel in job arguments or in middleware — and what did InfoTech's architecture get wrong?

solid border = note exists in your vault today · dashed = future entry, created when the pipeline sees you mention it again

• rev 12026-07-18 09:12 — page seeded from live assessment (4 probes, 2 blank)
• rev 209:19 — §2 rewritten around the GVL model + kitchen analogy
• rev 309:27 — InfoTech memory linked; §3 decoded; retrieval scheduled → 2026-07-21