Case study · in production

An AI assistant that learns from what it actually sold

Most AI chat projects end at the demo: it answers questions, everyone nods, nothing changes. This one is wired to the money. A visitor has a conversation, a human specialist closes the sale, the CRM records the margin — and that real outcome comes back to change what the agent knows next time. Four systems, one loop, built and owned by one person.

4 systems in the loop, built and owned solo

~96k hand-written lines of TypeScript across the estate

26 MCP tools on the knowledge platform, one route handler

89 HTTP route handlers across the three services I own

Every arrow is a real call path. The two that matter most are the last two: nothing reaches the agent without a human approving it, and once approved it takes effect without a deploy.

Before anything else

Why there is no logo on this page

The obvious question about an anonymous case study is whether it's real. Here is the honest answer, and the reason it reads the way it does.

The client has not agreed to be named yet, so they aren't. I've described the architecture, the decisions and the numbers in full, and left the brand out entirely. That's the right way round: their internals are theirs to publish, and a contractor who puts a client's system on a portfolio without asking is telling you exactly what they'll do with yours.

What that costs you as a reader is the ability to Google it. What it costs me is the logo. What you get instead is more detail than a named case study usually carries — because the identifying parts are the brand, not the engineering. On a call I'll screen-share the running systems, the dashboards and the commit history. If the client signs off later, the only thing that changes on this page is a name.

What each substitute stands for

On this page	Means	Detail
the client	A US travel company that sells complex multi-stop itineraries	Decades in business, a human sales team that closes on the phone, and a self-serve product alongside it. Named once they approve it.
the assistant	Their customer-facing chat agent on the public site	Collects a trip shape in conversation and hands qualified travellers to a human specialist.
the engine	The backend automation service	Reconciles every conversation to a desired end state. Owns the CRM writes, the emails, and the push into the brain.
the brain	The knowledge and learning platform	Retrieval, analysis of finished conversations, a human review queue, and versioned skills the agents load at runtime.
the CRM	Their order and trip system — the thing that actually sells	A 239-endpoint internal API. Holds the money: what was collected and what the margin was.
the self-serve builder	Their separate trip-building product on another subdomain	A different codebase, owned by a different engineer. Relevant because the identity gap between it and the main site is a real measurement problem.

Technology names are not anonymised — Next.js, Mastra, Postgres, pgvector and the rest are choices I made and can defend. Only the client's own product names are substituted.

The problem

Complex products don't sell themselves, and they don't qualify themselves either

The client sells itineraries with many moving parts. That means a real conversation before anyone can quote anything — and that conversation was consuming the time of the people who actually close.

Before

A specialist spent the first half of every inquiry establishing the basics — where, roughly when, how many people, what class.
Visitors who weren't ready to fill in a long form left with nothing, and were never heard from again.
Whatever the team learned in those conversations stayed in the conversations. Nothing accumulated.
And the analytics could not say which pages produced revenue, because the site and the point of sale were measured separately.

After

The agent handles qualification in conversation and hands over a trip shape, so the specialist starts at the interesting part.
A visitor who isn't ready can still leave an email against a specific route, on a durable path that can't drop it.
Every finished conversation is analysed against its real commercial outcome, and what's learned goes into a review queue.
Bookings carry an acquisition channel and a contribution margin, so "which channel earns" is a query rather than an argument.

The framing that mattered: the agent is not the product, it's the front of a funnel that ends in a human closing a sale. Everything about the design follows from that — which is why more than half of the work is reliability and measurement rather than prompting.

Screenshot pending

The chat surface, mid-conversation

The customer-facing side: transcript on one side, an editable route the visitor can reorder on the other, kept in two-way sync. The price shown is an estimate from the client's own engine and is labelled as one.

The shape

Four systems, three of them mine

Split at the boundaries that actually differ: what the customer touches, what a clock wakes up, what stores knowledge, and what already existed and holds the money.

The engine is the only service with no user interface — it exists to be woken by a clock and by a webhook. That separation is why a traffic spike on the public site cannot starve a retry sweep.

Public site + chat agent

The assistant

The only part a customer sees. It holds a real conversation, builds a trip shape as it goes, prices it from the client's own estimate engine, and hands the traveller to a human when the shape is good enough to be worth a specialist's time.

Two agents behind one chat surface, six typed tools between them
Retrieval is a tool call, not a prompt-stuffing step — the agent decides when it needs to look something up
1,465-line chat panel with two-way sync between the transcript and an editable route sidebar
Conversation memory in Postgres, so a returning visitor resumes rather than restarts

Next.js 16
React 19
Mastra
AI SDK
Neon
Sanity

Reconciliation + integrations

The engine

The unglamorous half. It does not fire on events and hope — it continuously compares every recent conversation against a desired end state and closes whatever gap it finds. Its own deploy target, its own secrets, so a burst of homepage traffic can never starve a retry sweep.

Five cron sweeps at four different cadences, from every 10 minutes to every 4 hours
Per-thread mutex so the webhook and the cron can never process the same conversation twice
Every step opens with a predicate that short-circuits if the invariant already holds
115 unit tests over the pure parts — payload mapping, normalisation, money parsing

Next.js 16
Workflow DevKit v4
Neon
SendGrid
PostHog

Knowledge, learning, review

The brain

Its own service, its own database, its own auth. It reads finished conversations, proposes changes to the agent's instructions, and lets a human approve or reject each one. Approved knowledge is published as a versioned bundle the agents load at runtime — no deploy needed to change agent behaviour.

One knowledge table spanning skills, Q&A, long-form wiki and imported source docs
Retrieval over-fetches, reranks on similarity plus observed usefulness, then slices to k
An audience column keeps internal-only rows — supplier margins, escalation paths — permanently out of the customer-facing agent's context
13 migrations, immutable version history on every knowledge entry, full audit log

Next.js 16
Drizzle
Postgres + pgvector
MCP
Neon Auth
Vercel Blob

The client's existing system — where money happens

The CRM

Not mine, and that is the point. A decades-old internal order system with a 239-endpoint API and no OpenAPI spec worth trusting. Integrating with it honestly — including documenting where its own fields lie — is most of what made the rest of the loop possible.

Two inbound webhooks consumed; the payload shape reverse-engineered from live deliveries and documented
Traveler count had to be derived from the passenger list — the obvious field undercounts, and the obvious sum overcounts
Orders carry what was collected and what the margin was; that is the only ground truth for whether any of this worked
A generated 239-endpoint catalogue now backs a read-only API console inside the brain admin

Read + webhook integration
OAuth2 / API keys
Idempotent consumers

The unglamorous half

The backend doesn't fire on events. It converges.

A handoff that silently fails is worse than one that never happened — the visitor believes a human is coming. So the automation is built as a controller: it declares what a finished conversation must look like and keeps closing the gap until it does.

The webhook exists for the human waiting on screen. The cron exists for the day the CRM is down. They run the same code, so there is one code path to reason about rather than a fast one and a correct one.

The invariants, as they're actually written

Needs doing when…	Action	Why it's safe to re-run
finished AND no CRM trip id	create the trip idea in the CRM	The id being set is the completion marker. Second run is a no-op.
finished AND no email timestamp	email the specialist	Timestamp written only on a confirmed success, so a failed send stays visible to the next sweep.
in progress AND idle > 60 min	mark abandoned	Terminal state. Cannot be entered twice.
all of the above satisfied	mark completed	Derived, not stored independently — it can't disagree with the parts.
completed AND settled > 4h AND ≥ 2 messages	push to the brain for analysis	Guarded by an ingested-at timestamp. The delay exists so analysis sees the final outcome.

Spam and test conversations are excluded from that last predicate. The learning corpus is the one place where a junk row costs more than it saves.

Retrieval

The agent looks things up. It cannot look everything up.

One knowledge base serves two readers with different rights: the customer-facing agent, and the team. Which makes the interesting question not 'how good is retrieval' but 'what stops the wrong reader from getting the wrong row'.

The audience value is hard-coded where the tool is called. It is not a parameter in the tool's input schema, so there is no argument the model can produce that widens its own access.

Design decisions worth naming

Retrieval is a tool, not a preprocessing step. The agent decides when it needs to look something up, so a simple turn costs one call instead of always paying for a search.
The embedding model is named in config, not in code. Both embedding tables carry the model that produced each vector, so swapping providers is a re-embed sweep rather than a migration nobody wants to start.
Rerank blends similarity with observed usefulness. A row that keeps proving relevant rises; a row that never does sinks without anyone curating it.
One table, four kinds. Short answers, long-form pages, imported documents and the agent's own instructions live together, so there is one retrieval path to make fast and secure instead of four.

The learning loop, precisely

A settled conversation is analysed against its real outcome — topics, objections, unanswered questions, why it ended.
A second pass turns that into proposed changes, each aimed at a specific named skill, with a confidence score.
New proposals are deduplicated against existing ones by embedding, so the twentieth report of the same gap increments a counter rather than filling the queue.
A human approves or rejects. This is a hard gate, not a default.
Approved changes are published as a versioned bundle the agents load at boot. The conversation records which version was live at the time, so the analysis knows what the agent actually knew.

Screenshot pending

The proposal review queue

What the analysis pipeline produces: proposed changes to a specific named skill, grouped by category, each with a confidence score and a diff against the current version. A human approves or rejects. Nothing ships unreviewed.

Screenshot pending

Skill editor with version history

Agent instructions are versioned per skill. A published bundle is a manifest pinning each skill to an exact version, content-hashed, so 'what was the agent running when this conversation happened' is answerable months later.

The platform also speaks MCP — 26 tools, 5 resources and 3 prompts on a single route handler, behind the same bearer auth as the REST API. The team's own AI clients drive the review queue through it, which means the tool that helps improve the agent is itself an agent interface. That was not a demo; it is how the queue gets worked.

Screenshot pending

MCP tool catalogue in the admin

The knowledge platform exposes 26 tools over MCP on a single route handler, with the same bearer auth as the REST API. The team's own AI clients drive the review queue through it instead of clicking.

Screenshot pending

Read-only API console over the client's CRM

A generated catalogue of 239 endpoints across 43 controllers, with a default-deny allow-list so only non-mutating calls are reachable from the browser. Built because 'what does this field actually return' was costing hours.

Measurement

A funnel nobody can trust is worse than no funnel

This is the half of the work that looks like marketing and is really engineering. An event contract written before launch, a reconciliation tolerance agreed with the client, and — the part I'm proudest of — catching the moment the dashboard started lying.

26% end-to-end conversion, opened chat → handed to a specialist

~15% real same-site conversion once scanner traffic is excluded

5–10× traffic inflation from link-scanner bots, found and neutralised

±5% reconciliation tolerance between site analytics and the CRM

First five days after launch. Small numbers, reported because they are the real ones — and because step 1 was later contaminated by scanner traffic, which is itself part of the story below.

The finding

Conversion wasn't 2–7%. It was about 15%.

For weeks the dashboard reported a same-site conversion rate that would have justified rebuilding the page. The page was fine. The denominator wasn't.

Nothing about the product changed. The number changed because the question changed from 'how many visitors' to 'how many visitors who were actually there'. The contaminated window was annotated rather than deleted, so the history stays readable.

Why this is the marketing skill, not the analytics chore. A wrong number does not sit still — it gets acted on. That 2–7% would have bought a redesign, or a discount, or a difficult conversation about whether the assistant was working. Finding the real ~15% did not require a new tool; it required not believing a chart. The durable output was a written rule for the whole team: never mix the two products in one person-level funnel, because identity continuity between them is about 0.6% — so any funnel spanning both would look precise and be nonsense.

Screenshot pending

The measurement dashboard

Eight named funnels, cohorts, and the engaged-visitor comparison that corrected the conversion rate. Every funnel definition is written down alongside the caveat that applies to it.

Revenue

Getting a marketing tag to survive all the way to contribution margin

The client's question was specific: of first-time individual bookings, what share came from one acquisition channel — and what share of margin did those represent? Answering it meant a tag surviving six hops through two systems I don't own.

The shortcut was to overwrite the CRM's existing origination field with the marketing source. It would have worked the same day and silently broken a different reconciliation that filters on that field.

What the booking event carries

Contribution margin — parsed from the CRM's own accounting block, not estimated
Booking type — individual or group, so the two never blend into one metric
First-time flag — tri-state, because "unknown" is not "no"
Acquisition channel — the marketing tag when present, the CRM's origination when not

Fired only on first insert, so a redelivered webhook cannot double-count a sale. The customer's email is the identity key and never a property — the raw address stays in the client's systems.

The trap I didn't take

The one-line version was to write the marketing source into the CRM's existing origination field. It would have shipped that afternoon and worked immediately.

It would also have broken a different metric, quietly: a separate reconciliation filters on exactly that field, so tagged bookings would have silently dropped out of its denominator — and nobody would have noticed for a month, at which point the two numbers disagree and neither is trustworthy.

So the tag went in a new field, which needed another engineer, in another codebase, on another schedule. Four deploys across four repositories, each designed as a no-op until the others land.

The hard parts

Five decisions where the obvious answer was wrong

Anyone can list technologies. What's worth reading is the point where the cheap option was available and I can tell you exactly what it would have cost.

Duplicate emails in production

What was wrong

A durable workflow retried a step that had already sent a confirmation email. The traveller got it twice. The retry was doing exactly what it was configured to do.

What I did

The email step now swallows the delivery error internally and returns success, so the workflow will not retry it inside the same run. The timestamp is written only on a confirmed 2xx, so the invariant stays honest — and the next sweep, minutes later, re-attempts a genuinely failed send.

Why the obvious answer was worse

The tempting fix is a 'sent' flag written before the call, which trades a duplicate for a silently lost email. That is strictly worse: a duplicate is visible and survivable, a lost handoff is neither. The residual risk — a provider timeout that actually delivered causing one duplicate on the next sweep — was named, costed, and accepted out loud rather than hidden.

A webhook and a cron racing on the same conversation

What was wrong

The user-facing path fires immediately so the traveller sees confirmation within seconds. The safety-net sweep runs every 15 minutes. Both call the same reconciliation, and they can overlap.

What I did

A per-conversation lease, taken as a single conditional update that only succeeds if no unexpired lease exists. Postgres makes the read and the write atomic in one statement, so two concurrent acquires cannot both win. The lease carries a 5-minute TTL, so a crashed worker releases its own lock instead of wedging the conversation forever.

Why the obvious answer was worse

An advisory lock would have been fewer lines and would not have survived the process dying mid-run. A queue would have been a new piece of infrastructure to operate for a problem that fits in one statement against a database that was already there.

An agent that could read data it should never see

What was wrong

The knowledge base holds both customer-facing answers and internal-only material — supplier margins, escalation paths. Both live in one table so that one retrieval path serves both readers.

What I did

The customer-facing agent's retrieval tool hard-codes its audience at the call site. It is not a parameter the model can set, and the filter is enforced in the SQL predicate, not in application code after the fetch.

Why the obvious answer was worse

The usual approach is to instruct the model not to reveal internal content. That is a request, not a boundary — one prompt injection and it is gone. The model here cannot ask for the rows in the first place, and the filter runs in the database, so a bug in the tool wrapper still cannot leak them.

The dashboard was reporting a conversion rate that was wrong by half

What was wrong

Same-site conversion read 2–7% and everyone believed it. It was not a product problem. Corporate link scanners had been inflating raw visitor counts 5–10× for weeks — in one sample week roughly 92% of 'visitors' fired a single pageview and nothing else.

What I did

The scanners cannot be dropped at ingestion — they present as real desktop browsers and no signal exists at pageview time. So the metric changed instead: count only visitors who fired a page-leave event, which a scanner never does. That cohort is flat across the contamination window, and it reads the real ~15%.

Why the obvious answer was worse

Everyone's instinct is a user-agent blocklist. It does not work here and, worse, it looks like it works — you get a smaller number and stop asking. Changing the definition of the metric to something bots structurally cannot fake is bot-resistant by construction rather than by maintenance. Contaminated windows were annotated rather than deleted, so the history stays readable.

Attributing contribution margin across four codebases and two engineers

What was wrong

The client wanted to know what share of first-time individual bookings came from a specific acquisition channel, and what share of margin those represented. The answer required a tag to survive from a marketing link, through a product I do not own, through a CRM I do not own, to an analytics event.

What I did

A six-hop chain: capture the tag on first load before the router strips it, attach it to the submission only when non-empty, have the platform engineer persist it as a separate field on the trip idea and add it to the order webhook, then map it onto a server-side booking event carrying margin, booking type and a first-time flag. The booking event fires only on first insert, so a webhook redelivery cannot double-count.

Why the obvious answer was worse

The shortcut was to overwrite the CRM's existing origination field with the marketing source. It would have worked immediately and quietly broken the reconciliation that another metric depended on, because that one filters on origination. Four separate deploys across four repositories, each a no-op until the others land — so deploy order does not matter and nothing breaks in between.

The commercial outcome

What it earned — the part I won't guess at

This is the section a case study normally opens with, and normally invents. The instrumentation to answer it is live and firing; I have not yet sat down with the client's CRM and read the numbers out. So the tiles below are empty, and each one says exactly which report fills it.

— revenue booked through the loop Pending

— contribution margin attributed to a channel Pending

— qualified inquiries per week through the assistant Pending

— specialist minutes saved per qualified inquiry Pending

4 of 4 commercial numbers are not filled in, and I'd rather show you the gap than a plausible figure. Every other number on this page was counted from a repository or read out of a live analytics project. These four come from the client's CRM, which means reading their revenue data and getting their agreement before publishing it. Until both happen, they stay blank.

If you're evaluating me and this is the section you care about most — fair. Ask and I'll walk you through the live dashboards on a call, which is more convincing than a number on a page you'd have to take on trust anyway.

Where every number on this page came from

Figure	Source
4 systems in the loop, built and owned solo	Chat agent, automation service, knowledge platform, plus the integration into their existing CRM. Three are deploy targets I own end to end.
~96k hand-written lines of TypeScript across the estate	Counted 2026-07-26: 629 .ts/.tsx files, 110,816 lines total, of which ~15k is a generated API catalogue (239 endpoints, 43 controllers). Hand-written remainder ≈ 96k.
26 MCP tools on the knowledge platform, one route handler	Counted from the MCP server's tool registrations, plus 5 resources and 3 prompts. Conversations, proposals, skills, bundles, knowledge CRUD, retrieval, and two analytics tools.
89 HTTP route handlers across the three services I own	23 on the public app, 22 on the automation service, 44 on the knowledge platform.
26% end-to-end conversion, opened chat → handed to a specialist	Launch-week baseline: 23 opened → 15 engaged → 10 trip shape complete → 7 one click away → 6 handed off. Small sample, first five days. Reported as a baseline, not a steady state.
~15% real same-site conversion once scanner traffic is excluded	The dashboard read 2–7% on raw pageviews. Filtering to visitors who fired a page-leave event — which scanners never do — gives ~15%. Conversion was never the problem; the denominator was.
5–10× traffic inflation from link-scanner bots, found and neutralised	In one sample week ~92% of 'visitors' fired a single pageview and nothing else. Not filterable at ingestion — the fix is a query-time engagement cohort.
±5% reconciliation tolerance between site analytics and the CRM	The definition-of-done on the measurement sprint: instrumentation had to agree with the CRM's own lead count within ±5% before any of it was allowed to inform a decision.
pending revenue booked through the loop	Pending: CRM orders graph, `collected[]` summed over the window, filtered to orders whose trip idea originated in the assistant or the on-site capture path.
pending contribution margin attributed to a channel	Pending: group the server-side booking event by `source_channel` and sum `contribution_margin`. The event ships and fires; the numbers just have not accumulated into a report yet.
pending qualified inquiries per week through the assistant	Pending: count finished threads with a CRM trip-idea id, weekly. The instrumentation is live — this is a dashboard read, not new work.
pending specialist minutes saved per qualified inquiry	Pending: needs the client's own before/after estimate of qualifying a cold inquiry by hand. Will be labelled as their estimate, not a measurement.

What this isn't

The limits, said out loud

Same reason the skills page lists what I don't do: a boundary you find here costs you ten seconds, and a boundary you find three emails into a project costs considerably more.

It is not autonomous

Every proposed change to agent behaviour goes into a human review queue. Nothing the analysis pipeline produces reaches a customer without someone approving it. That was a design requirement, not a limitation.

I did not train a model

This is applied LLM engineering — agents, tools, retrieval, structured output, and the reliability layer underneath. No fine-tuning, no training runs. If you need someone to train a model, that is a different person.

The revenue attribution is partly outside my control

Two of the six hops in the attribution chain live in systems owned by the client's platform engineer. My half is deployed. The end-to-end number lights up when theirs is.

Cross-product identity is still broken and I said so

Person continuity between the main site and the self-serve product measures about 0.6%. Rather than build a funnel that spans both and quietly lies, the rule is written down: never put a step from each in the same person-level funnel. Cross-product volume is tracked a different way.

One route-capture surface is still behind a flag

It shipped as a canary on a fraction of pages, not a full rollout. The measurement and the durable lead path were the load-bearing parts of that sprint; the broad rollout was deliberately deferred rather than rushed.

Agent observability is the thinnest part

Per-turn token cost and tool latency are emitted by the framework but not yet forwarded into the analytics project. It is on the list, it is not done, and I would rather write that here than have you find it.

FAQ

The questions this architecture raises

Roughly in the order a technical reader asks them — including whether the loop is honestly closed.

Why three separate apps instead of one?

Because they fail differently and they must not be able to starve each other. The public app serves visitors and a burst of homepage traffic must never delay a retry sweep, so the backend is its own deploy target with its own secrets and its own schedule.

The knowledge platform is separate again because it owns its own database and its own migrations. Boundaries that exist only as a comment get crossed; boundaries that are separate deploy targets do not.

Why does a human have to approve everything the agent learns?

Because the alternative is a system that changes what it tells customers without anyone deciding that it should. The analysis pipeline reads finished conversations and proposes changes to the agent's instructions; a person approves or rejects each one, and publishing is a second, separate action.

That was a design requirement rather than a limitation. The value is in the proposal — the loop noticing that the same objection keeps arriving — not in the deployment being automatic.

Is it honestly a closed loop, then?

Not in the fully automatic sense, and I would rather name it than let the title imply otherwise. Approve, then a second click to publish, then the agents pick up the new bundle on their next cold start. Called plainly, it is a well-instrumented suggestion box with a manual deploy step.

What makes it worth building anyway is that every part of it is real and inspectable: conversations are analysed against their actual commercial outcome, proposals are deduplicated so the same idea does not fill the queue twenty times, and every published bundle is versioned so “what was the agent running when this conversation happened” is answerable months later.

What actually happens when something in the chain fails?

The system converges rather than retries blindly. It repeatedly compares every recent conversation against what a finished one is supposed to look like and closes whatever gap it finds, so a failed step is late rather than lost — and each step opens with a check that short-circuits if the work is already done.

The honest guarantee on the emails is at most once per attempt and at least once overall, not exactly once. The two windows where a duplicate is still possible are named in the code, next to the date of the incident that taught us about them.

Does this shape transfer to a product that isn't travel?

The domain does not; the shape does. Wherever a product is complex enough that a form cannot qualify a buyer, you get the same three problems: a conversation that collects a real shape, a reliability layer that guarantees the handoff, and measurement that proves any of it worked.

The parts that would be rebuilt are the qualification rules and the integration into whatever holds your money. The reconciliation pattern and the review-gated learning loop move across almost unchanged.

Next step

If you have a product that needs a conversation before it sells

This shape transfers. The agent, the reliability layer underneath it, and the measurement that proves it worked are the same three problems wherever the product is complex enough that a form can't qualify a buyer.

Book a walkthrough of the real thing See the stack, with evidence

A walkthrough is a screen-share of the running systems and the dashboards behind these numbers — not a slide deck.