Awesome Process-First Concurrency

Reach for OS processes and simple queues before actors or a distributed message bus. Justify the complexity.

A curated, decision-guide awesome-list on getting real concurrency out of the primitives your OS already gives you — processes, pipes, cron, and a queue table — before reaching for an actor runtime or a Kafka cluster.

Contents

• Introduction
• Decision Heuristics
• Process-Based Parallelism
• Simple Job & Queue Mechanisms
• Message Brokers at "Just Enough" Scale
• Actor Systems — and When They're the Right Call
• Further Reading
• When You Actually Need an Actor System or Distributed Bus
• Contributing
• License

Introduction

Unix's oldest idea is still one of its best: small, single-purpose processes, composed with pipes and files, each one independently schedulable by a kernel that has been load-balancing processes across cores for decades. A shell pipeline, a pool of worker processes reading off a directory of files, or a handful of cron jobs writing to a lock file will comfortably carry a huge fraction of real-world background-work and fan-out workloads.

Yet it's common to reach straight for an actor framework or a Kafka topic for problems that a xargs -P invocation or a Redis list would solve in an afternoon. Actor systems and distributed message buses solve real problems — location transparency, back-pressure across a fleet of machines, exactly-once delivery semantics at scale — but they also import real costs: new failure modes, new operational surface, and a mental model (supervision trees, partitioning, consumer groups) that a process pool never asks you to learn.

This list is about the boring, load-bearing option: OS processes and simple queues first. Escalate to actors or a distributed bus only once you have a concrete reason — not because it's what the tutorial started with.

Licensing note: every resource below is tagged with its license/status. Two entries are worth reading closely before you adopt them: Akka's relicensing to BUSL 1.1 in 2022 (source-available, not OSI-approved open source above a revenue threshold — see Apache Pekko for the FOSS fork), and Redis's relicensing history (RSAL/SSPL from 2024, back to AGPL-3.0 for Redis 8 — see Valkey if you want a permissively-licensed drop-in). A "process-first, justify the complexity" reader should apply the same scrutiny to a dependency's license as to its operational cost.

Decision Heuristics

Rough thresholds for "processes are still fine" vs. "you've outgrown them":

• Single machine, bounded fan-out (tens to low thousands of concurrent units of work) → a process pool or xargs -P is enough. You don't need a scheduler that spans machines to schedule work on one machine.
• Work items are independent and idempotent-ish → a simple queue table (even SQLite) with a SELECT ... WHERE claimed_at IS NULL claim pattern gets you most of what a message broker gives you, without a broker to operate.
• You need cross-machine fan-out → this is the first real signal you need something beyond a single box's process table — but that alone doesn't mean you need Kafka; a lightweight broker like NATS core often bridges the gap.
• You need in-order, partitioned, replayable delivery at high sustained throughput (event sourcing, stream processing, multiple independent consumer groups replaying history) → this is genuinely Kafka/Pulsar territory.
• Your unit of concurrency needs to hold long-lived, mutable, addressable state and receive messages over its lifetime (a game session, a per-user connection, a saga/workflow instance) → this is what actor systems are actually for. A process pool re-derives state from scratch per task; actors don't.
• You're reaching for actors or a bus mainly because of language/library hype, not a concrete scaling or state-modeling problem you've hit → that's the signal to stay with processes and queues longer.

Process-Based Parallelism

• GNU Parallel (GPL-3.0) — build and run parallel pipelines from the shell; handles job control, retries, and remote execution over SSH with one tool.
• xargs -P (GPL, GNU findutils) — the simplest possible parallel-process runner; already on every Unix box.
• Python multiprocessing (PSF License, stdlib) — process-based parallelism with a Pool API, sidesteps the GIL for CPU-bound work.
• Joblib (BSD-3-Clause) — lightweight parallel-for helper built on multiprocessing/loky, popular in the Python data/scientific stack.
• concurrent.futures.ProcessPoolExecutor (PSF License, stdlib) — standard-library process pool with a futures-style API.
• Ray (Apache-2.0) — the natural next step once you've genuinely outgrown a single machine's process pool and need distributed task/actor scheduling across a cluster.

Simple Job & Queue Mechanisms

• cron + a lock file (flock) (open source, part of most *nix base systems) — the original scheduled-job system; still correct and still running most servers' background work.
• systemd timers (LGPL-2.1) — cron's more observable modern sibling, with logging and dependency ordering for free on systemd hosts.
• Redis Lists / Streams used as a plain queue (AGPL-3.0 as of Redis 8 — see licensing note below) — LPUSH/BRPOP or XADD/XREADGROUP give you a durable, shared queue without standing up a broker cluster.
• Valkey (BSD-3-Clause) — the Linux Foundation-backed, unambiguously permissive-FOSS fork of Redis (forked at the pre-relicensing codebase); a drop-in replacement if you want the same list/stream commands without tracking Redis's license history.
• SQLite as a queue (e.g. litequeue) (MIT) — a transactional claim-and-process queue backed by a single file; no server process to run at all.
• Procrastinate (MIT) — a Python task queue built directly on PostgreSQL, no separate broker.
• pg-boss (MIT) — the same idea for Node.js: job queue on top of Postgres.

Message Brokers at "Just Enough" Scale

• NATS (core pub/sub) (Apache-2.0, CNCF) — a single small binary, sub-millisecond latency, no ZooKeeper/controller cluster to run — often the right stop between "a Redis list" and "a Kafka cluster."
• Redis Streams with consumer groups (AGPL-3.0 as of Redis 8; use [Valkey](https://valkey.io/) (BSD-3-Clause) if you want the same feature under a permissive license) — durable, ordered, multi-consumer semantics without a separate broker technology if you already run Redis.
• Kafka (Apache-2.0, ASF) — reach for this once you need partitioned, replayable, high-throughput log semantics with multiple independent consumer groups; it is a real distributed system to operate, not a drop-in queue.
• Apache Pulsar (Apache-2.0, ASF) — Kafka-adjacent, with built-in multi-tenancy and tiered storage; same "you need this much" bar applies.

Actor Systems — and When They're the Right Call

• Erlang/OTP (Apache-2.0) — the origin of the actor/supervision-tree model, built for systems that must keep running while individual processes crash and restart.
• Elixir (Apache-2.0) — Erlang/OTP's ergonomics on the BEAM; a common modern entry point to the actor model.
• Apache Pekko (Apache-2.0, ASF) — a genuine open-source actor toolkit for Scala/Java on the JVM, forked from Akka's pre-relicensing (2.6.x) codebase and now an Apache Software Foundation project. Prefer this over Akka if you need an unambiguously FOSS-licensed JVM actor runtime.
• Akka (JVM) (BUSL 1.1 — source-available, NOT OSI-approved open source since v2.7/Sept 2022) — mature and still widely deployed, but production use beyond the license's free tier requires a commercial agreement with Akka's maintainer (Akka Inc., formerly Lightbend). Listed for completeness; see Apache Pekko above for the FOSS equivalent.
• Microsoft Orleans (MIT) — the "virtual actor" model: actors are always addressable and the runtime handles activation/placement for you.
• Proto.Actor (Apache-2.0) — a lighter cross-language (Go/C#/Kotlin) actor library if you want the model without the BEAM or JVM.

Further Reading

• Rob Pike, "Concurrency is not Parallelism" — the talk that made the process/goroutine vs. parallelism distinction mainstream; useful for separating "I want things to happen concurrently" from "I need more throughput."
• The Unix philosophy as described in The Art of Unix Programming (Eric S. Raymond) — the "do one thing, compose via pipes" design ethic underlying process-first concurrency.
• Pat Helland, "Life Beyond Distributed Transactions" — good background on why distributed messaging designs get complicated once you actually need cross-node guarantees.

When You Actually Need an Actor System or Distributed Bus

Be honest about the signal, not the hype:

• You have long-lived, addressable, stateful units of work (per-session, per-device, per-workflow) that need to receive messages over time — actors model this far better than re-hydrating state from a database on every task.
• You need partitioned, replayable, ordered delivery to multiple independent consumer groups at sustained high throughput — that's Kafka/Pulsar's actual job.
• You need cross-machine fault tolerance with supervision — a process crashing on one box and a queue message needing to be picked up by a different box, automatically, with retry/backoff policy — actor runtimes and brokers both do this natively; a bare process pool does not.
• You're past the point where "a bigger box and more worker processes" is a real option — genuine horizontal scale-out needs, not anticipated ones.

If none of those apply yet, a process pool and a queue table will very likely outlast your assumptions about needing more.

Contributing

Contributions welcome! Please read CONTRIBUTING.md first — in short: resources should be real, actively relevant to the process-first-concurrency thesis, added in the right category in alphabetical order, and free of self-promotion spam. Open an issue with the suggest a resource template or send a PR directly.

License

To the extent possible under law, the contributors have waived all copyright and related or neighboring rights to this work. See LICENSE (CC0 1.0 Universal).