Case study · AI systems · Multi-agent · Operations
A multi-agent AI platform, run like production
My own AI operating environment, which I call the Forge. It routes each task to a local or cloud model by how sensitive the data is, runs agent work on a schedule across two machines and watches itself for the kind of failure that doesn't throw an error.
RoleArchitect, sole developer
Running sinceMarch 2026
Scheduled jobs~30 across two machines
Agents19
01 · The problem
A chat window forgets everything and checks nothing
Most day-to-day AI use happens in a chat window. Context resets every session, nothing runs while you're asleep and nobody checks whether the answer was right. I wanted the other version: an environment that keeps what it learns, runs real work on a schedule, sends each task to the right model and tells me when something breaks.
It handles several kinds of work with different handling rules, so some tasks may use a cloud model and some have to stay on hardware I own. Getting that right, and proving it stays right, turned out to be most of the engineering.
02 · Model routing
Every task goes to a model by sensitivity, and restricted work fails closed
Tasks are classified before dispatch. Summaries, classification, extraction and formatting run on a local model on a 12GB consumer GPU at no API cost. Code, architecture and hard reasoning go to Claude. 11 task types route to local models, and 4 restricted tiers refuse to run if their local model is down instead of falling back to a cloud model.
A fallback chain that can move work to a less private model without telling anyone fails at the exact moment the protection matters. So on a restricted tier an outage is an error, never a reason to downgrade, and no task can opt itself out of its own restriction.
03 · Operations
The part a demo never shows: monitoring that tells the truth
Building the agents was the easy half. The harder half was finding out how often a system can look healthy while doing nothing useful. Three cases did the most to change how I build.
01
A model throwing away most of its input with no error
The local model server defaulted to a 4K context window and silently truncated anything longer. No error, just answers built from the tail end of the document, for months. In a controlled test on a 75K-token input, an 8K-context model found 0 of 4 target facts and the corrected 128K configuration found 4 of 4. Health checks now catch truncation directly.
02
A backup check that the clock could satisfy
One backup was judged fresh by its archive's filename, which a timestamp regenerated every night whether or not the backup inside worked. It reported healthy for twelve days while the database inside was frozen. The fix reads the modification time of the database inside the archive, which only a working backup can advance.
03
A machine can't report its own silence
Heartbeats and alerts that run on the box being watched go quiet at the same moment it does. A watchdog on a separate machine now covers 8 backup jobs and tells three states apart: failed, stale and never reported. Each machine checks the other, and neither is trusted to certify itself.
The rule that came out of it
An untested control is a hypothesis. Every boundary and alarm gets checked by attempting the thing it should stop, with throwaway data so a failed test isn't itself the breach, and every alert is verified by reading it back at its destination rather than trusting a return code.
04 · Architecture
Context, memory, scheduled agents and routing
Context
A bootstrap under 300 lines
Identity, active projects, agent roster and session protocol load in under a minute. Everything else is modular and pulled in on demand, so the first task doesn't start with half the context window already spent.
modular ruleson-demand load
Memory
Version-controlled and model-portable
Facts, decisions and discovered quirks persist across sessions as plain markdown in git, with no dependence on a provider's memory feature. A finding gets recorded once and never rediscovered.
markdowngitcross-session
Scheduled agents
Explicit exits and a circuit breaker
Unattended agent runs work through multi-step tasks with a done condition on every phase, so there are no open-ended loops. Three consecutive failures on the same call halts the run and writes a blocked checkpoint. Nothing commits until a person approves it.
Simple, high-volume work stays on the local GPU and the cost of every cloud call is logged per task. The model is treated as a component with a budget.
OllamaClaudecost logging
05 · What runs on it
Jobs that run whether or not I'm at the keyboard
When
Job
6:30 every morning
Briefing assembled from project state and posted to Slack with no manual input
Nightly
Research pipeline that pulls transcripts from YouTube channels I follow, extracts what's useful and files it into a searchable knowledge base
On demand
Meeting transcription with Whisper on the local GPU, so recordings never leave the network (35 meetings so far)
Always on
A Slack bot answering from the local model and escalating to Claude only for hard requests
Nightly
Encrypted backups to local and off-site storage, checked from a second machine
06 · What this shows
The skills that matter once AI leaves the demo
Specification
Every agent file and task is a spec that has to run with no one to ask at 2am. Ambiguity at spec time becomes a wrong build overnight.
Evaluation
A two-line approach statement before execution catches wrong directions cheaply, and each catch becomes a written rule so it doesn't repeat.
Orchestration
An orchestrator calls workers and agents never call each other, with handoffs, exit conditions and a circuit breaker on every run.
Failure diagnosis
Each failure is sorted first: a bad instruction, a model reasoning error or a tool error. Each one has a different fix, and debugging the wrong category wastes the session.
Data-sensitivity design
Routing by sensitivity that fails closed, secrets kept out of tool-server environments and controls proven by test instead of assumed.
Context and cost
Research subagents hand back summaries instead of raw file dumps, the bootstrap stays small and local models absorb the high-volume work.