Back to projects
Distributed Systems2026

Durable Agent Engine

Fault-Tolerant LLM Agent Orchestrator


Durable Agent Engine screenshot 1

Overview

Most agent demos show the happy path. This one shows what happens when a worker crashes mid-step, two workers race for the same job, or a tool call fails — and proves the run still finishes correctly, exactly once, every time.

Technical Details

  • Built a Postgres-native durable queue for multi-step LLM agent DAGs — no Redis, no queue library — with SKIP LOCKED exactly-once claiming, a lock-TTL reaper that reclaims work from crashed workers, and exponential-backoff retries into a dead-letter table.

  • Proved crash-safety with a chaos test injecting 40% failures plus kill -9 mid-run: every run still reaches a terminal state, with an append-only audit log of every state transition.

  • Live demo streams each step's claim, retry, and completion over WebSockets so the failure handling is observable rather than asserted.

Tech Stack

Node.jsPostgreSQLWebSocketsFly.io