CI workers, queues, and distributed execution

Context

Continuous integration looks like a button in GitHub. Underneath it is a distributed system: events from a forge, work queued for machines, runners with a lifecycle, logs that must be durable, and failures that must be classified (user test failed vs. infrastructure ate the job).

Founding-engineer work at MonkCI sat in that problem space: making execution reliable enough that developers can ignore the machinery.

Problem

The difficult parts are not the happy path. They are: a worker that dies mid-job, a retry that double-starts a non-idempotent step, an event that arrives twice, a runner that is busy but not healthy, a timeout that is indistinguishable from a hung test.

CI multiplies these issues across untrusted workloads, bursty queues, and a user population that correctly treats infrastructure flakes as the platform’s bug, not theirs.

Constraints

Jobs must be isolatable. One workload cannot poison the next.

Event delivery from the forge is at-least-once until you prove otherwise.

Users need a story for every terminal state: passed, failed, cancelled, timed out, infrastructure error.

Observability has to explain a single job without requiring a distributed-tracing hobby project on day one — but you still need enough signal to debug races.

Architecture

Forge events enter an ingestion path, are persisted, and become queue work. Workers claim jobs, provision or assign a runner, stream logs, and report terminal state. Failures either retry with policy or dead-letter with a reason a human can read. Runner lifecycle (acquire, idle, recycle, drain) is explicit, not implied by “the process is still up.”

Engineering decisions

Persist the event before acting on it. Lost events are worse than delayed jobs.

Distinguish application failure from infrastructure failure in the state machine. Mixing them trains users to re-run everything.

Retries need a budget and a key. Infinite retry with a fresh runner can hide a poison payload.

Go is a natural fit for this class of service: explicit concurrency, straightforward deployment, good fit for workers and proxies. That is a supporting fact, not the point of the work.

Tradeoffs

A fat worker that does scheduling, execution, and log shipping is simpler until you need to scale one axis independently.

Aggressive timeouts keep queues moving and create false failures. Loose timeouts hide wedged runners. The policy has to be visible and tunable.

At-least-once execution plus non-idempotent user scripts is an unsolved user-facing problem; the platform can only make retries obvious and rare.

Failure modes

Split brain: two workers claim the same job.

Retry storms after a downstream outage.

Log loss on worker crash, leaving a failed job with no evidence.

Idle runners that look available and are not.

Validation

Fault injection on workers and queues: kill a worker mid-job and assert the job lands in a defined state.

Idempotency tests on event ingestion.

Classification tests: a failing user test is not reported as an infrastructure incident.

Lessons

Reliability is a state machine plus evidence. If you cannot explain why a job died, you do not operate a CI system — you operate a lottery.

Queues and retries are not implementation details. They are the product’s behavior under stress.