Case study  ·  AI systems  ·  Multi-agent  ·  Operations

A multi-agent AI platform, run like production

My own AI operating environment, which I call the Forge. It routes each task to a local or cloud model by how sensitive the data is, runs agent work on a schedule across two machines and watches itself for the kind of failure that doesn't throw an error.

RoleArchitect, sole developer
Running sinceMarch 2026
Scheduled jobs~30 across two machines
Agents19

A chat window forgets everything and checks nothing

Most day-to-day AI use happens in a chat window. Context resets every session, nothing runs while you're asleep and nobody checks whether the answer was right. I wanted the other version: an environment that keeps what it learns, runs real work on a schedule, sends each task to the right model and tells me when something breaks.

It handles several kinds of work with different handling rules, so some tasks may use a cloud model and some have to stay on hardware I own. Getting that right, and proving it stays right, turned out to be most of the engineering.

Every task goes to a model by sensitivity, and restricted work fails closed

Tasks are classified before dispatch. Summaries, classification, extraction and formatting run on a local model on a 12GB consumer GPU at no API cost. Code, architecture and hard reasoning go to Claude. 11 task types route to local models, and 4 restricted tiers refuse to run if their local model is down instead of falling back to a cloud model.

A fallback chain that can move work to a less private model without telling anyone fails at the exact moment the protection matters. So on a restricted tier an outage is an error, never a reason to downgrade, and no task can opt itself out of its own restriction.

The part a demo never shows: monitoring that tells the truth

Building the agents was the easy half. The harder half was finding out how often a system can look healthy while doing nothing useful. Three cases did the most to change how I build.

01
A model throwing away most of its input with no error
The local model server defaulted to a 4K context window and silently truncated anything longer. No error, just answers built from the tail end of the document, for months. In a controlled test on a 75K-token input, an 8K-context model found 0 of 4 target facts and the corrected 128K configuration found 4 of 4. Health checks now catch truncation directly.
02
A backup check that the clock could satisfy
One backup was judged fresh by its archive's filename, which a timestamp regenerated every night whether or not the backup inside worked. It reported healthy for twelve days while the database inside was frozen. The fix reads the modification time of the database inside the archive, which only a working backup can advance.
03
A machine can't report its own silence
Heartbeats and alerts that run on the box being watched go quiet at the same moment it does. A watchdog on a separate machine now covers 8 backup jobs and tells three states apart: failed, stale and never reported. Each machine checks the other, and neither is trusted to certify itself.
The rule that came out of it

An untested control is a hypothesis. Every boundary and alarm gets checked by attempting the thing it should stop, with throwaway data so a failed test isn't itself the breach, and every alert is verified by reading it back at its destination rather than trusting a return code.

Context, memory, scheduled agents and routing

Context
A bootstrap under 300 lines
Identity, active projects, agent roster and session protocol load in under a minute. Everything else is modular and pulled in on demand, so the first task doesn't start with half the context window already spent.
modular ruleson-demand load
Memory
Version-controlled and model-portable
Facts, decisions and discovered quirks persist across sessions as plain markdown in git, with no dependence on a provider's memory feature. A finding gets recorded once and never rediscovered.
markdowngitcross-session
Scheduled agents
Explicit exits and a circuit breaker
Unattended agent runs work through multi-step tasks with a done condition on every phase, so there are no open-ended loops. Three consecutive failures on the same call halts the run and writes a blocked checkpoint. Nothing commits until a person approves it.
orchestrator → workerscircuit breakerhuman approval
Routing
Local first where quality allows
Simple, high-volume work stays on the local GPU and the cost of every cloud call is logged per task. The model is treated as a component with a budget.
OllamaClaudecost logging

Jobs that run whether or not I'm at the keyboard

WhenJob
6:30 every morningBriefing assembled from project state and posted to Slack with no manual input
NightlyResearch pipeline that pulls transcripts from YouTube channels I follow, extracts what's useful and files it into a searchable knowledge base
On demandMeeting transcription with Whisper on the local GPU, so recordings never leave the network (35 meetings so far)
Always onA Slack bot answering from the local model and escalating to Claude only for hard requests
NightlyEncrypted backups to local and off-site storage, checked from a second machine

The skills that matter once AI leaves the demo

Specification

  • Every agent file and task is a spec that has to run with no one to ask at 2am. Ambiguity at spec time becomes a wrong build overnight.

Evaluation

  • A two-line approach statement before execution catches wrong directions cheaply, and each catch becomes a written rule so it doesn't repeat.

Orchestration

  • An orchestrator calls workers and agents never call each other, with handoffs, exit conditions and a circuit breaker on every run.

Failure diagnosis

  • Each failure is sorted first: a bad instruction, a model reasoning error or a tool error. Each one has a different fix, and debugging the wrong category wastes the session.

Data-sensitivity design

  • Routing by sensitivity that fails closed, secrets kept out of tool-server environments and controls proven by test instead of assumed.

Context and cost

  • Research subagents hand back summaries instead of raw file dumps, the bootstrap stays small and local models absorb the high-volume work.

Hybrid local and cloud, on hardware I own

Agent framework
Claude Code
Local models
Ollama / Gemma
Language
Python
Speech
Whisper
Messaging
Slack API
Scheduling
cron / systemd / Task Scheduler
GPU
RTX 3060 12GB
OS
Linux + Windows