# The closed loop > The architecture — a chat agent, the reliability layer under it, and a knowledge platform that learns from finished conversations. - Canonical: https://iambogdan.com/case-studies/closed-loop - Author: Bogdan Andronic (AI Engineer & Full-Stack TypeScript Developer) - Source: generated from the data modules this page renders - Records: 31 - Part of: A US travel company (Built for a client · anonymised) ## Measured ### systems in the loop, built and owned solo Chat agent, automation service, knowledge platform, plus the integration into their existing CRM. Three are deploy targets I own end to end. - value: 4 ### hand-written lines of TypeScript across the estate Counted 2026-07-26: 629 .ts/.tsx files, 110,816 lines total, of which ~15k is a generated API catalogue (239 endpoints, 43 controllers). Hand-written remainder ≈ 96k. - value: ~96k ### MCP tools on the knowledge platform, one route handler Counted from the MCP server's tool registrations, plus 5 resources and 3 prompts. Conversations, proposals, skills, bundles, knowledge CRUD, retrieval, and two analytics tools. - value: 26 ### HTTP route handlers across the three services I own 23 on the public app, 22 on the automation service, 44 on the knowledge platform. - value: 89 ### end-to-end conversion, opened chat → handed to a specialist Launch-week baseline: 23 opened → 15 engaged → 10 trip shape complete → 7 one click away → 6 handed off. Small sample, first five days. Reported as a baseline, not a steady state. - value: 26% ### real same-site conversion once scanner traffic is excluded The dashboard read 2–7% on raw pageviews. Filtering to visitors who fired a page-leave event — which scanners never do — gives ~15%. Conversion was never the problem; the denominator was. - value: ~15% ### traffic inflation from link-scanner bots, found and neutralised In one sample week ~92% of 'visitors' fired a single pageview and nothing else. Not filterable at ingestion — the fix is a query-time engagement cohort. - value: 5–10× ### reconciliation tolerance between site analytics and the CRM The definition-of-done on the measurement sprint: instrumentation had to agree with the CRM's own lead count within ±5% before any of it was allowed to inform a decision. - value: ±5% ## The systems ### The assistant The only part a customer sees. It holds a real conversation, builds a trip shape as it goes, prices it from the client's own estimate engine, and hands the traveller to a human when the shape is good enough to be worth a specialist's time. - role: Public site + chat agent - facts: Two agents behind one chat surface, six typed tools between them · Retrieval is a tool call, not a prompt-stuffing step — the agent decides when it needs to look something up · 1,465-line chat panel with two-way sync between the transcript and an editable route sidebar · Conversation memory in Postgres, so a returning visitor resumes rather than restarts - stack: Next.js 16 · React 19 · Mastra · AI SDK · Neon · Sanity ### The engine The unglamorous half. It does not fire on events and hope — it continuously compares every recent conversation against a desired end state and closes whatever gap it finds. Its own deploy target, its own secrets, so a burst of homepage traffic can never starve a retry sweep. - role: Reconciliation + integrations - facts: Five cron sweeps at four different cadences, from every 10 minutes to every 4 hours · Per-thread mutex so the webhook and the cron can never process the same conversation twice · Every step opens with a predicate that short-circuits if the invariant already holds · 115 unit tests over the pure parts — payload mapping, normalisation, money parsing - stack: Next.js 16 · Workflow DevKit v4 · Neon · SendGrid · PostHog ### The brain Its own service, its own database, its own auth. It reads finished conversations, proposes changes to the agent's instructions, and lets a human approve or reject each one. Approved knowledge is published as a versioned bundle the agents load at runtime — no deploy needed to change agent behaviour. - role: Knowledge, learning, review - facts: One knowledge table spanning skills, Q&A, long-form wiki and imported source docs · Retrieval over-fetches, reranks on similarity plus observed usefulness, then slices to k · An audience column keeps internal-only rows — supplier margins, escalation paths — permanently out of the customer-facing agent's context · 13 migrations, immutable version history on every knowledge entry, full audit log - stack: Next.js 16 · Drizzle · Postgres + pgvector · MCP · Neon Auth · Vercel Blob ### The CRM Not mine, and that is the point. A decades-old internal order system with a 239-endpoint API and no OpenAPI spec worth trusting. Integrating with it honestly — including documenting where its own fields lie — is most of what made the rest of the loop possible. - role: The client's existing system — where money happens - facts: Two inbound webhooks consumed; the payload shape reverse-engineered from live deliveries and documented · Traveler count had to be derived from the passenger list — the obvious field undercounts, and the obvious sum overcounts · Orders carry what was collected and what the margin was; that is the only ground truth for whether any of this worked · A generated 239-endpoint catalogue now backs a read-only API console inside the brain admin - stack: Read + webhook integration · OAuth2 / API keys · Idempotent consumers ## Projects ### The Brain An MCP server exposing 26 typed tools, 5 resources and 3 prompts over streamable HTTP, sitting on the same business layer as the REST API — conversations, proposals, skills, bundles, knowledge CRUD, retrieval and two analytics tools, so the team's own AI clients work the review queue through it instead of clicking. Retrieval is pgvector cosine with over-fetch and a custom rerank, and it returns citations. New knowledge arrives through a two-step LLM pipeline on a SKIP LOCKED queue, gets embedded, and is deduplicated against what is already known. Approved knowledge is published as an immutable, hash-manifested bundle the agents load at runtime. - kind: Flagship · production - summary: Two AI agents share one brain. It holds what the company knows, answers with citations, and takes in new knowledge through a review pipeline — so I can change how both agents behave without shipping a deploy. - stack: Next.js 16 · Mastra · AI SDK v6 · MCP SDK · Drizzle · Postgres + pgvector · Neon Auth - status: live ### Dual-agent travel assistant Six custom tools, Postgres-backed memory, and single-turn lane routing decided from the agent's own tool calls rather than a classifier round-trip. On the front end: a large React 19 chat panel with two-way chat/sidebar sync, a complete product funnel, and race-hardened persistence — a JSONB merge UPSERT that fixed a real first-write race in production. - kind: Production · customer-facing - summary: One chat window, two specialized agents behind it, each picking up the parts of the conversation that belong to it. When the brain feeding them goes down, the chat keeps working anyway. - stack: Next.js 16 · Mastra · AI SDK v6 · React 19 + Compiler · Neon · PostHog - status: live ### Crash-safe automation backend A per-thread reconciliation engine on a durable workflow runtime, with idempotent steps and retry at the cron level rather than inside the step. The mutex is an atomic JSONB compare-and-swap with a recovery TTL, so a crashed worker releases its own lock. Lead capture has the full durability contract — honeypot, rate limit, idempotency key, pending-before-third-party write, 30-day dedupe, retry sweep. Production alerting throttles across three layers and folds duplicates by fingerprint. - kind: Production · the plumbing - summary: Kill the server halfway through a job and nothing gets sent twice, nothing gets lost, and the lock releases itself. Nobody puts this in a demo. It's the reason everything above it stays up. - stack: Next.js 16 · Workflow DevKit v4 · Neon · OAuth2 integrations · SendGrid - status: live ## The hard parts ### Duplicate emails in production A durable workflow retried a step that had already sent a confirmation email. The traveller got it twice. The retry was doing exactly what it was configured to do. - fix: The email step now swallows the delivery error internally and returns success, so the workflow will not retry it inside the same run. The timestamp is written only on a confirmed 2xx, so the invariant stays honest — and the next sweep, minutes later, re-attempts a genuinely failed send. - why: The tempting fix is a 'sent' flag written before the call, which trades a duplicate for a silently lost email. That is strictly worse: a duplicate is visible and survivable, a lost handoff is neither. The residual risk — a provider timeout that actually delivered causing one duplicate on the next sweep — was named, costed, and accepted out loud rather than hidden. ### A webhook and a cron racing on the same conversation The user-facing path fires immediately so the traveller sees confirmation within seconds. The safety-net sweep runs every 15 minutes. Both call the same reconciliation, and they can overlap. - fix: A per-conversation lease, taken as a single conditional update that only succeeds if no unexpired lease exists. Postgres makes the read and the write atomic in one statement, so two concurrent acquires cannot both win. The lease carries a 5-minute TTL, so a crashed worker releases its own lock instead of wedging the conversation forever. - why: An advisory lock would have been fewer lines and would not have survived the process dying mid-run. A queue would have been a new piece of infrastructure to operate for a problem that fits in one statement against a database that was already there. ### An agent that could read data it should never see The knowledge base holds both customer-facing answers and internal-only material — supplier margins, escalation paths. Both live in one table so that one retrieval path serves both readers. - fix: The customer-facing agent's retrieval tool hard-codes its audience at the call site. It is not a parameter the model can set, and the filter is enforced in the SQL predicate, not in application code after the fetch. - why: The usual approach is to instruct the model not to reveal internal content. That is a request, not a boundary — one prompt injection and it is gone. The model here cannot ask for the rows in the first place, and the filter runs in the database, so a bug in the tool wrapper still cannot leak them. ### The dashboard was reporting a conversion rate that was wrong by half Same-site conversion read 2–7% and everyone believed it. It was not a product problem. Corporate link scanners had been inflating raw visitor counts 5–10× for weeks — in one sample week roughly 92% of 'visitors' fired a single pageview and nothing else. - fix: The scanners cannot be dropped at ingestion — they present as real desktop browsers and no signal exists at pageview time. So the metric changed instead: count only visitors who fired a page-leave event, which a scanner never does. That cohort is flat across the contamination window, and it reads the real ~15%. - why: Everyone's instinct is a user-agent blocklist. It does not work here and, worse, it looks like it works — you get a smaller number and stop asking. Changing the definition of the metric to something bots structurally cannot fake is bot-resistant by construction rather than by maintenance. Contaminated windows were annotated rather than deleted, so the history stays readable. ### Attributing contribution margin across four codebases and two engineers The client wanted to know what share of first-time individual bookings came from a specific acquisition channel, and what share of margin those represented. The answer required a tag to survive from a marketing link, through a product I do not own, through a CRM I do not own, to an analytics event. - fix: A six-hop chain: capture the tag on first load before the router strips it, attach it to the submission only when non-empty, have the platform engineer persist it as a separate field on the trip idea and add it to the order webhook, then map it onto a server-side booking event carrying margin, booking type and a first-time flag. The booking event fires only on first insert, so a webhook redelivery cannot double-count. - why: The shortcut was to overwrite the CRM's existing origination field with the marketing source. It would have worked immediately and quietly broken the reconciliation that another metric depended on, because that one filters on origination. Four separate deploys across four repositories, each a no-op until the others land — so deploy order does not matter and nothing breaks in between. ## What this does not claim ### It is not autonomous Every proposed change to agent behaviour goes into a human review queue. Nothing the analysis pipeline produces reaches a customer without someone approving it. That was a design requirement, not a limitation. ### I did not train a model This is applied LLM engineering — agents, tools, retrieval, structured output, and the reliability layer underneath. No fine-tuning, no training runs. If you need someone to train a model, that is a different person. ### The revenue attribution is partly outside my control Two of the six hops in the attribution chain live in systems owned by the client's platform engineer. My half is deployed. The end-to-end number lights up when theirs is. ### Cross-product identity is still broken and I said so Person continuity between the main site and the self-serve product measures about 0.6%. Rather than build a funnel that spans both and quietly lies, the rule is written down: never put a step from each in the same person-level funnel. Cross-product volume is tracked a different way. ### One route-capture surface is still behind a flag It shipped as a canary on a fraction of pages, not a full rollout. The measurement and the durable lead path were the load-bearing parts of that sprint; the broad rollout was deliberately deferred rather than rushed. ### Agent observability is the thinnest part Per-turn token cost and tool latency are emitted by the framework but not yet forwarded into the analytics project. It is on the list, it is not done, and I would rather write that here than have you find it. ## Questions and answers ### Why three separate apps instead of one? Because they fail differently and they must not be able to starve each other. The public app serves visitors and a burst of homepage traffic must never delay a retry sweep, so the backend is its own deploy target with its own secrets and its own schedule. The knowledge platform is separate again because it owns its own database and its own migrations. Boundaries that exist only as a comment get crossed; boundaries that are separate deploy targets do not. ### Why does a human have to approve everything the agent learns? Because the alternative is a system that changes what it tells customers without anyone deciding that it should. The analysis pipeline reads finished conversations and proposes changes to the agent's instructions; a person approves or rejects each one, and publishing is a second, separate action. That was a design requirement rather than a limitation. The value is in the proposal — the loop noticing that the same objection keeps arriving — not in the deployment being automatic. ### Is it honestly a closed loop, then? Not in the fully automatic sense, and I would rather name it than let the title imply otherwise. Approve, then a second click to publish, then the agents pick up the new bundle on their next cold start. Called plainly, it is a well-instrumented suggestion box with a manual deploy step. What makes it worth building anyway is that every part of it is real and inspectable: conversations are analysed against their actual commercial outcome, proposals are deduplicated so the same idea does not fill the queue twenty times, and every published bundle is versioned so “what was the agent running when this conversation happened” is answerable months later. ### What actually happens when something in the chain fails? The system converges rather than retries blindly. It repeatedly compares every recent conversation against what a finished one is supposed to look like and closes whatever gap it finds, so a failed step is late rather than lost — and each step opens with a check that short-circuits if the work is already done. The honest guarantee on the emails is at most once per attempt and at least once overall, not exactly once. The two windows where a duplicate is still possible are named in the code, next to the date of the incident that taught us about them. ### Does this shape transfer to a product that isn't travel? The domain does not; the shape does. Wherever a product is complex enough that a form cannot qualify a buyer, you get the same three problems: a conversation that collects a real shape, a reliability layer that guarantees the handoff, and measurement that proves any of it worked. The parts that would be rebuilt are the qualification rules and the integration into whatever holds your money. The reconciliation pattern and the review-gated learning loop move across almost unchanged.