Field device ticket classifier with a reasoning cross-check
An AI routing service for a state transportation agency's field maintenance tickets. It runs unattended for $5 to $10 a month in API costs, and before it assigns anything it checks whether the model's reasoning actually agrees with the model's answer.
RoleSolutions architect, sole developer
ClientState transportation agency
In productionSince early 2026
API cost$5 to $10 / month at ~100 tickets a day
01 · The problem
A 15-minute routing window that manual checks kept missing
The agency maintains thousands of roadside devices (cameras, message signs, detection sensors, ramp meters) across hundreds of miles of highway. When one fails a ticket opens in the service desk, and it has to reach the right team within 15 business-hour minutes: electrical, communications, software or vendor escalation.
The catch was downstream. The maintenance contractor's system only picks up a ticket once it has a device record attached, so an unrouted ticket was invisible to it while the clock kept running. Nobody found out until a field response was already late.
People were checking the queue about every 10 minutes, each. The window was 15. That margin doesn't survive a busy shift, a lunch break or a weekend on call, so the system had to poll faster than any person and route before the window closed.
02 · The key decision
The cheapest AI call is the one you don't make
The obvious build was to send every ticket through the model. The better question was what the ticket already knew before the model ever saw it.
Operators could pick the device type and routing group from a dropdown when they opened a ticket. If they had, the answer was already there and there was nothing to guess. Checking that field first meant about 85% of tickets never reached the classifier at all.
The routing decision
Poll every 2 minutes → operator already picked a device? Assign it, no API call → otherwise classify with Claude Haiku and keep its reasoning → pull the routing signal out of that reasoning with regex → compare it to the declared answer → a match assigns, a mismatch goes to a human review queue with the reasoning attached.
03 · Two models, two jobs
The right model for each task
Classifier
Claude Haiku
Runs on every unassigned ticket on a 2-minute cycle. Picked for speed and cost: at about 100 tickets a day the bill stays between $5 and $10 a month. The prompt asks for reasoning before the decision, and that reasoning is what the cross-check reads.
~100 tickets/day · $5 to $10/mo · $20 hard cap
Narrative drafts
Claude Sonnet
Drafts the formal explanations for long-running delays in the monthly compliance report. That text goes to contract leadership, so writing quality matters more than cost on a once-a-month batch.
Monthly batch · a manager edits and approves every draft
04 · Architecture
Three small services on one Linux host
Service 1
Poller and classifier
APScheduler runs every 2 minutes, pulls unassigned tickets from the service desk REST API with a service-account token, checks the operator's pre-selection, then routes or classifies and writes the result to SQLite.
APSchedulerREST APISQLite
Service 2
Pickup monitor
Watches for the contractor system's confirmation on each routed ticket and records the business-hours time from creation to pickup. That's the number the response requirement is measured on.
business-hours calcSLA tracking
Service 3
Dashboard
A Flask app with the human review queue (mismatches with the full reasoning trace), compliance metrics, monthly report generation and the screen where a manager edits and locks the delay narratives.
Flaskreview queueExcel export
Validation layer
Reasoning cross-check
Regex reads routing keywords out of Haiku's reasoning and works out which queue that reasoning points to. If it disagrees with the queue the model declared, the ticket goes to a person. This catches the model that reasons its way to the right answer and then outputs a different one.
regexcoherence checkaudit trail
05 · Pipeline
From a new ticket to the right team in under 2 minutes
01
Poll the service desk every 2 minutes
Replaced a manual check roughly every 10 minutes per person.
02
Deterministic gate on the operator's pick
If the device dropdown is filled in, assign it with no model involved. About 85% of tickets leave the pipeline here.
03
Classify the rest with reasoning kept
The remaining ~15% go to Claude Haiku. The reasoning text is saved, because it's the input to the next step.
04
Cross-check reasoning against the answer
Above a 70% confidence threshold with a matching signal, the ticket is assigned. Low confidence or any disagreement sends it to the review queue.
05
Confirm pickup and record the time
The monitor waits for the contractor system's confirmation and logs the real business-hours transfer time.
06
Monthly report with drafted narratives
The dashboard builds the compliance report with an exclusion audit trail (aborted, duplicate, merged and cancelled tickets, each with a logged reason). Sonnet drafts delay explanations and a manager edits and locks them before they go out.
06 · Results
Cheap to run, fast to route and built to be left alone
85%
Routed with no model call
Operator picks assigned directly. The model only sees the tickets that actually need it.
8.7 min
Average time to pickup
Business hours, ticket creation to contractor pickup, against a 15-minute window.
$5 to $10
Monthly API cost
With a $20 hard cap. A small fraction of the penalty exposure it protects.
Test mode before production
All new AI routing started in test mode with every ticket sent to review
Auto-assign was switched on only after 50+ tickets at 90% accuracy or better
Same rule for the narratives: the model drafts and a person approves before anything is locked
Low maintenance on purpose
systemd services, SQLite and one Linux host, with no infrastructure beyond the API
The same collector, scheduler, SQLite and Flask pattern was reused for other monitoring on the contract
One codebase and one deployment for all of it
07 · What this shows
The failure label-only benchmarks miss
Most classifier evaluations grade the output label against a labeled test set. That measurement is needed, and it has a blind spot. A model whose reasoning points at the right queue while its declared answer points somewhere else will pass any test that only looks at the label.
On a 15-minute window with penalties attached, a silent misroute costs real money. The cross-check is one regex pass. The review queue hands the uncertain cases to a person with the reasoning attached, so no failure goes unseen.
I've carried the same rule into every classification system I've built since: the model classifies, regex checks the reasoning and any mismatch goes to a human. Escalation is in the design from day one instead of being added after the first incident.