Self-learning proxy debugger

See the full writeup: I built a CI/CD pipeline that fixes itself — and remembers how. This page is the artefact card; the note is the architecture in depth.

What

A debugging and self-healing pipeline that watches the proxy fleet for failures, matches known ones against a vector database of previously verified fixes, and — for genuinely novel failures — asks a local LLM to draft a fix from RAG-retrieved context. Every proposed change flows through the same CI/CD, dual-signal QA gate, and blue-green traffic switch a human fix would, so production never sees an unverified patch. When a fix holds, it’s written back into the vector database with commit-hash-anchored metadata; the next occurrence of the same failure class gets resolved by the fast path with no LLM call at all.

Runs entirely on-prem on a Mac Studio M3 Ultra (256GB). No API bills.

Why

The proxy platform I lead at Vanguard runs ~10M requests/day across ~50 nodes, and the failure classes recur. Once-a-week bugs eat one-a-day engineer time — investigate, reproduce, patch, verify, forget. The motivating question was: what’s the smallest system that would resolve 90% of recurring failures without human touch, while making genuinely new failures cheaper to investigate? The answer turned out to be less about the LLM and more about how to keep learned knowledge from rotting.

How

Six components, all built to be replaceable:

• Blue-green deployment — every change lands on the idle environment first. Production is structurally incapable of receiving unverified code. Rollback is a traffic switch, not a redeploy.
• Dual-signal QA gate — a custom YOLO model for visual regression on the UI, plus a functional smoke-test suite for the failures a vision model would wave through (API errors, data corruption). Both must pass before traffic switches.
• Fix matching — two-stage. A cheap error-signature fingerprint (exception type + file + stack trace hash) pre-filters candidates, then embedding similarity runs on the survivors. Pure embedding search on raw error messages is the classic RAG trap — the fingerprint stage is what stops it.
• Fast path vs LLM path — trusted cached fix → apply directly, no LLM call. Miss / stale / quarantined → local LLM generates a fix from RAG-retrieved history. Three failed attempts → escalate. The LLM cannot modify its own verifiers; an automation loop that can edit its own judges isn’t self-healing, it’s self-deceiving.
• Event-driven staleness — every learned fix carries a status (trusted / stale / quarantined) that transitions on real events, not on a clock. Git hook diffs a changed file → any fix referencing it downgrades. A cached fix fails in CI → quarantined on the spot. Commit-hash metadata means reviewing a stale fix means reading a diff rather than reconstructing context.
• Hermes → Telegram approval loop — complex fixes route to a Telegram bot with Approve/Reject buttons. Deliberately runs on a separate cloud VM, not the Mac Studio, so the escalation channel can’t share fate with the machine it’s escalating about.

The economics compound: every LLM call is an investment that eliminates future LLM calls. Cache hit rate climbs; the expensive path shrinks to genuinely novel problems.

What was hard (and what I actually learned)

The interesting engineering wasn’t the LLM — it was knowledge staleness. Every auto-learning system I looked at either (a) never invalidated its knowledge, and slowly poisoned itself as the underlying codebase drifted, or (b) invalidated everything periodically and threw away most of the value it accumulated. Neither is right.

Event-driven, commit-hash-anchored invalidation is the version that works in practice. Downgrade fixes when the files they reference change; keep old-but-still-correct fixes indefinitely; ask the LLM to audit only when retrieval actually surfaces a stale candidate. It also happens to be my answer to the knowledge-conflict problem that started this whole journey — most conflicts in a learning system aren’t fix-versus-fix, they’re old-fix-versus-new-reality.

The other thing that shaped the design: the escalation channel cannot share fate with the thing it’s monitoring. Hermes lives off-box for the same reason your smoke detector is battery-powered.

Honest limitations

• No shadow-mode graduation yet. A newly-learned fix goes straight to trusted on approval — cheaper to run in shadow (LLM + cache in parallel, agreement earns promotion) but that’s not built.
• No vector-DB backups or embedding versioning. The fix cache is the least reproducible asset in the system — code can be rebuilt, months of learned fixes can’t. Swapping embedding models without a re-index silently breaks every similarity match.
• Single-machine runtime. Local LLM inference on one Mac Studio = no failover if the box goes down. Fine for the current workload; not a strategy at 10× the load.
• No canary traffic routing. Traffic switches at 100% after QA passes. Some bugs only appear under real load, and a 5–10% canary step would catch them without gambling all of production.
• The vector DB is not a public artefact. Day-job data; I can’t open-source what’s in it. The pattern is the reusable part — documented in the full writeup.

What’s next

In rough order:

1. Conflict detection at learn-time — before storing a new fix, check for existing fixes touching the same files and lines; escalate overlaps to Telegram. Preventing contradictory knowledge at write-time beats resolving it at retrieval-time.
2. Shadow-mode graduation for newly-learned fixes.
3. Canary traffic — 5–10% first, watch the watchdog, then commit.
4. Replay-based eval harness. The vector DB is already a labelled dataset of real production failures; any change to the LLM, its prompt, or the retrieval logic should be benchmarked against that replay set before it ships.
5. Vector-DB backups + embedding versioning.

Related

• Full architecture writeup: I built a CI/CD pipeline that fixes itself — and remembers how.
• The layer this sits on top of: Edge routing proxy (10M req/day proxy platform) and the paused Rotation service (health + load balancing across the fleet).