I built a CI/CD pipeline that fixes itself — and remembers how

How a Mac Studio, a local LLM, a YOLO model, and a vector database learned to keep my deployments healthy without me.

Project page: Self-learning proxy debugger — status, tags, and the artefact-card view of what this note describes in depth.

It started with hallucinations

Like everyone running a local LLM, I hit the hallucination problem early. RAG solved most of it — ground the model in retrieved context and it stops inventing things.

But RAG introduced two new problems that nobody warns you about: knowledge conflicts (retrieved documents disagreeing with each other, or with the model’s own training) and good old AI flattery — the model agreeing with whatever framing I gave it, even when the sources said otherwise.

Fixing those forced me to think much harder about what enters my knowledge base and when it stops being true. That thinking eventually grew into something bigger than a chatbot: a deployment pipeline that detects its own failures, fixes them, verifies the fixes, and — this is the part I’m most proud of — learns from every fix so it never has to solve the same problem twice.

Everything runs locally on a Mac Studio M3 Ultra with 256GB of memory. No API bills. Here’s how it works.

The core loop

The pipeline has one job: keep production healthy with as little of my attention as possible.

1. Deploy. Every change goes through CI/CD into a blue-green setup — new code lands on the idle environment, never directly on live traffic.
2. Verify. A QA gate runs against the idle environment. It’s dual-signal: a YOLO model I trained myself does visual regression on the UI, and a functional smoke-test suite catches the non-visual failures (API errors, data corruption) that a vision model would happily wave through.
3. Switch. Only when both signals pass does traffic move to the new environment. A watchdog service then monitors runtime health — because some bugs only show up hours after deployment.
4. Heal. When something fails, the interesting part begins.

flowchart TD
    subgraph DEPLOY["Blue-Green Deployment"]
        A[Code Change / Fix] --> B[CI/CD Pipeline]
        B --> C[Deploy to Idle Env<br/>green]
        C --> D[QA Gate:<br/>YOLO Visual Tests<br/>+ Functional/API Smoke Tests]
        D -->|Pass| E[Traffic Switch<br/>green → live]
        D -->|Fail| FAIL[No switch —<br/>production untouched]
    end

    E --> W[Watchdog Service<br/>runtime health monitoring]
    W -->|Healthy| DONE([Stable])
    W -->|Error within N hours| RB[Instant Rollback<br/>switch traffic back to blue]
    RB --> QRT[Quarantine the fix]

    FAIL --> MATCH
    W -->|New error detected| MATCH

    subgraph MATCHING["Fix Matching"]
        MATCH[Error Signature Fingerprint<br/>exception type + file + stack trace] --> EMB[Embedding Similarity<br/>on pre-filtered candidates]
        EMB --> Q{Match found?}
    end

    subgraph CACHE["Fast Path (no LLM)"]
        Q -->|Yes — trusted| H[Apply Cached Fix]
    end

    subgraph LLMPATH["LLM Path"]
        Q -->|No / stale / quarantined| I[Local LLM]
        R[(RAG Vector DB<br/>fixes + metadata:<br/>datetime · file · commit hash<br/>status · hit/fail counts)] -->|Retrieve context| I
        I --> J[Generate Fix]
    end

    H --> B
    J --> B
    QRT --> R

    D -->|"Fail 3x"| PAGE([Escalate to Human])

    E --> CPLX{Complexity<br/>Evaluation}
    CPLX -->|Trivial| LEARN[Learn fix into Vector DB<br/>+ metadata & commit hash]
    CPLX -->|Complex| HERMES

    subgraph NOTIFY["Hermes Notification Layer"]
        HERMES[Hermes Middleman] --> TG[Telegram Bot]
        TG -->|"Fix diff + commit hash<br/>Approve / Reject buttons"| HUMAN[Human on Mobile]
        HUMAN -->|Approve| LEARN
        HUMAN -->|Reject| QRT
        HUMAN -.->|"No response in 24h<br/>→ stays quarantined"| QRT
        W -->|Metrics digest<br/>rate-limited| TG
        PAGE --> TG
    end

    LEARN --> R

    subgraph MAINT["Cache Maintenance"]
        T1[File changed<br/>git hook diff] --> S{Status Update}
        T2[Cached fix failed] --> S
        T3[Age + low usage] --> S
        S -->|trusted / stale / quarantined| R
        AUDIT[Periodic LLM Audit<br/>batch review of old fixes<br/>vs current codebase] --> S
        METRICS[Metrics Dashboard<br/>cache hit rate · fix success rate<br/>escalation rate] -.-> AUDIT
    end

    R -.-> METRICS

The interesting part: fixes that compound

When a failure is detected, the system first asks: have I seen this before?

Matching happens in two stages. First, a cheap, precise error-signature fingerprint — exception type, file name, stack trace hash — pre-filters candidates. Only then does embedding similarity run on the survivors. This two-stage design matters: pure embedding search is the classic RAG trap where a superficially similar error message retrieves a confidently wrong fix.

If a trusted fix exists, it’s applied directly — no LLM call at all. It goes through the exact same CI/CD, QA gate, and blue-green verification as everything else. Known failures get resolved in the time it takes to run the pipeline, essentially for free.

If nothing matches, the local LLM is invoked with the failure context plus relevant history retrieved from the vector database. It generates a fix, which flows through the same gates. There’s a hard cap — three failed attempts and the system stops trying and pages me instead. And the LLM is deliberately scoped so it can’t touch its own verifiers: it cannot “fix” the QA tester, the watchdog, or the pipeline config. An automation loop that can modify its own judges isn’t self-healing; it’s self-deceiving.

When a fix succeeds, a complexity evaluator scores it. Trivial fixes are learned into the vector database immediately. Complex ones require my approval first — more on that in a second. Either way, once learned, that failure class is handled by the fast path forever after.

The economics of this compound beautifully: every LLM call is an investment that eliminates future LLM calls. Over time, the cache hit rate climbs and the expensive path shrinks to genuinely novel problems.

Trust decays, and the system knows it

Here’s the failure mode that kills most auto-learning systems: a fix learned six months ago gets replayed against a codebase that has since changed underneath it — and because the fast path skips the LLM, nothing sanity-checks it.

So every learned fix carries metadata — timestamp, referenced files, the commit hash at learn-time, hit and failure counts — and a status: trusted, stale, or quarantined.

• Trusted fixes apply automatically.
• Stale fixes apply, but need explicit watchdog confirmation and get flagged for review.
• Quarantined fixes never auto-apply; the system treats them as if they were never learned.

Status changes are event-driven, not periodic. A git hook diffs changed files against fix metadata — touch a file a fix references, and that fix is downgraded. A cached fix that fails in CI is quarantined on the spot and evicted after a second failure. Old fixes that haven’t been hit in months get reviewed on next use. And a periodic batch job has the LLM audit old trusted fixes against the current codebase — far cheaper than checking at apply-time.

The commit hash turns out to be the most valuable field. A timestamp tells me a fix is old; the hash tells me exactly what changed since, so reviewing a stale fix means reading a diff instead of reconstructing context from scratch.

This is also, incidentally, my answer to the knowledge-conflict problem that started this whole journey. Most conflicts in a learning system aren’t fix-versus-fix — they’re old-fix-versus-new-reality. Staleness tracking attacks the root cause.

Approving fixes from my phone

A human gate is only as good as its friction. If approving a fix requires opening a laptop, fixes sit in the queue for hours and the gate becomes the bottleneck — or worse, I start rubber-stamping.

So complex fixes route through Hermes, a small middleman service, to a Telegram bot. The message contains the diff, the commit hash, the complexity score, and Approve/Reject buttons. I can review a fix properly from anywhere. No response within 24 hours? The fix stays quarantined and the system falls back to the LLM path — safe by default, never blocked.

Hermes also delivers watchdog metric digests (rate-limited — alert fatigue is how monitoring dies) and escalation pages when the retry cap is hit.

One architectural decision worth calling out: Hermes does not run on the Mac Studio. Your escalation channel cannot share fate with the machine it’s escalating about. If the Mac hangs, everything on it dies silently — including the service whose only job is to tell me something died. Hermes lives on a small always-on cloud VM, which also runs an external heartbeat check: if the Mac goes quiet, I hear about it. I considered a spot VM with an auto-restart trick (Cloud Logging catches the preemption event, triggers Cloud Run, which restarts the instance — a pattern I use elsewhere and love), but for the alerting lifeline specifically, even a few minutes of recovery gap is the wrong trade. Ten dollars a month of boring reliability wins.

Why blue-green changed the safety story

Originally I planned snapshot-and-restore rollbacks. Blue-green made that obsolete: since every fix — human or machine — deploys to the idle environment first, production is structurally incapable of receiving unverified code. And rollback becomes an instant traffic switch. If the watchdog detects trouble within N hours of a switch, traffic flips back and the offending fix is quarantined automatically.

The one place this gets tricky is the data layer — an instant traffic switch doesn’t undo a schema migration. The expand/contract pattern (make schema changes backward-compatible in one deploy, remove old structures in a later one) is the answer, and it’s worth adopting before you need it, not after.

What’s next

The core loop works. Here’s what’s on the roadmap, roughly in order:

• Conflict detection at learn-time. Before storing a new fix, check for existing fixes touching the same files and lines; on overlap, escalate a merge/supersede decision to Telegram. Preventing contradictory knowledge at write-time beats resolving it at retrieval-time.
• Shadow mode for graduating fixes. A fix’s first few replays run alongside a parallel LLM call; agreement earns promotion to trusted. Cheap insurance against the cache learning something subtly wrong.
• Canary traffic. Shift 5–10% of traffic first, watch the watchdog, then commit. Some bugs only appear under real load.
• A replay-based eval harness. The vector DB is already a labeled dataset of real production failures. Any change to the LLM, its prompt, or the retrieval logic should be benchmarked against that replay set before it ships.
• Vector DB backups and embedding versioning. The fix cache is now the least reproducible asset in the system — the code can be rebuilt; months of learned fixes can’t. And swapping embedding models without a re-index silently breaks every similarity match.

The takeaway

The pattern underneath all of this is simple: automation you can trust needs three things — verification it can’t bypass, knowledge that expires, and a human gate with near-zero friction. Get those right and the system genuinely compounds: every failure it survives makes the next one cheaper.

Get any of them wrong and you’ve built something worse than manual ops — an automation that confidently repeats its old mistakes at machine speed.

Built with: a Mac Studio M3 Ultra (256GB), a local LLM, a custom-trained YOLO model, a vector database, blue-green CI/CD, and a Telegram bot. Questions or war stories from your own self-healing setups? hi@zhen.ee.