Case study · analytics & measurement

The analytics layer that tells you whether the AI actually worked

Building the product and proving it worked turn out to be two different jobs, and I do both. This is the measurement layer under a client's rebuilt site: an event contract agreed before the code, eight named funnels, dashboards the team reads daily, and metrics their own AI can query in plain language. The finding that justified all of it: five to ten times the recorded audience was corporate link scanners — and no ingestion filter could remove them.

46 events pinned by a written contract before the code

8 named funnels, each published with the caveat that limits it

5–10× traffic inflation from scanner bots, found and neutralised

2 read-only analytics tools an LLM agent can actually call

This is the half most people selling "AI engineer" can't do. Anyone can put a number on a dashboard. The work is deciding whether that number is allowed to inform a decision — because a wrong one does not sit quietly. It buys a redesign, kills a channel, or settles an argument it had no business settling.

Before anything else

Why there's no logo on this page either

Same client as the closed-loop write-up, same rule. They haven't agreed to be named, so they aren't — and on a page full of their internal metrics that matters more than usual.

Their numbers are theirs to publish. What's here is the method and the figures that describe the measurement rather than the business — engagement rates, contract counts, a conversion correction. No revenue, no absolute lead volume, no account identifiers.

That last one is the leak nobody checks: a live analytics project id next to a console URL is a working handle on an unnamed client's account. Dashboard ids, cohort ids and ticket numbers are stripped for the same reason. Even the event names are described rather than quoted — the client's product names are embedded in their own event prefixes.

What each substitute stands for

On this page	Means	Detail
the client	A US travel company that sells complex multi-stop itineraries	Same client as the closed-loop write-up. Named once they approve it.
the main site	Their public marketing site, rebuilt and relaunched in June 2026	The rebuild is the event everything here is measured across — and the thing I was asked to evaluate, having helped build it.
the self-serve builder	Their trip-building product, on a separate subdomain	Different codebase, different engineer, untouched by the rebuild. The identity boundary between it and the main site is the most expensive measurement problem here.
the assistant	The customer-facing chat agent on the main site	The thing the analytics exist to evaluate. Its own write-up is the closed-loop page.
the brain	Their internal knowledge platform	Where the funnel definitions and their caveats live, and where the team's own AI reads them from.
the client CRM	Their order and lead system — the system of record	Decades old and not mine. It holds what was actually sold, which is what any analytics claim eventually has to answer to.

Vendor technology is not anonymised. Naming Microsoft SafeLinks as the scanner is what makes the next section concrete rather than hand-wavy, and it isn't the client's information.

The finding

The traffic was real. The visitors weren't.

Corporate mail security follows every link in every email that passes through a company mailbox. Those requests arrive as ordinary desktop Chrome with a search-engine referrer, fire exactly one pageview, and leave. From late June they were most of the recorded audience — and the reported conversion rate was wrong by more than half as a result.

600–960 engaged visitors per week — flat straight across the rebuild

~15% vs 2–7% same-site conversion, correctly denominated versus bot-diluted

1.2% vs 15.5% desktop versus mobile CTA rate — the tell that the desktop bulk wasn't human

1.5% how far the obvious bot fix actually moved the number

Sixteen weeks, April to mid-July 2026. The engaged series is every week, measured. The raw series is drawn only at its four documented readings and dashed between them, because I do not have the weeks in between and will not draw a line through numbers I did not read. Two vertical scales, because on one scale the flat series is a line on the axis — which is the finding, but not a readable chart.

Everything obvious, and why it failed

The vendor's own bot filter. On by default, and it misses these entirely — they present as a real browser, because a real browser is what they are.
Drop them at ingestion. Impossible. At the moment a pageview fires there is no client-side signal separating a scanner from a person who is about to bounce.
A user-agent blocklist. Worse than useless here, because it appears to work: you get a smaller number and stop asking questions.
Exclude a cohort of confirmed scanners. Built it. All of them together moved a 10,232-visitor week to 10,077 — only a sliver are identifiable that way.

What worked: change the metric, not the filter

Count people who fire a page-leave event. A single-pageview scanner never does. Nothing to maintain, nothing to keep up to date.
The swap was cheap, and that was checked first. That event carries the same host, path, device-type and referring-domain properties, so every existing breakdown survived intact.
Applied at query time, not at ingestion. The contaminated history stays in the database and stays readable — annotated rather than deleted.
The prose had to move with the numbers. The internal page carried an authored bot-inflation narrative; fixing the figures and leaving the copy would have made it contradict itself.

The tell, if you want to catch this on your own dashboard. Pageviews per visitor collapsed from about 2.4 to 1.1 against a 2.2–2.6 norm, and desktop was converting at 1.2% against mobile's 15.5% in the same product. A thirteen-fold device gap is not a device gap. Both were sitting on the dashboard for weeks before anyone thought to look, because the headline number was going up.

Drawn this way on purpose. The instinct on seeing 2–7% become 15% is to read it as a win, and it is not one — it is the same people clicking the same button, counted against a denominator that had been inflated for weeks. A conversion rate is a claim about two numbers, and only one of them was ever in question.

Screenshot pending

The real-versus-inflated visitor insight

The chart that settled it: two weekly series on one axis, engaged flat underneath while raw climbs away. Built as a saved insight rather than a one-off query, so the claim stays checkable by someone who wasn't in the conversation.

What got built

Write the events down before the code, or the code becomes the spec

46 events and 147 typed property slots, agreed before anything was wired. Then eight named funnels on top of them — each one published with the specific way it can mislead you, because a definition without its caveat gets read six months later by someone who wasn't in the room.

What the contract fixes per event

Required versus optional, explicitly. 123 of the 147 slots are required, so a breakdown on them can never silently lose rows to a missing property.
84 names for 147 slots. The reuse is the point — the same name means the same thing on every event that carries it, which is what makes cross-event analysis legal rather than plausible.
A double-count guard where it matters. Once per thread, once per pageview, once per section, or an idempotency key. The cheapest way to corrupt a funnel is an impression event that re-fires when someone scrolls back up.
A PII level, per event. Email is an identity key and never a property, so raw addresses stay inside the client's systems.
Server-side where it matters. Lead lifecycle steps are emitted by the backend, so an ad-blocker can shorten a funnel's first step but never its outcome.

One prohibition outranks the rest

Never put a step from the main site and a step from the self-serve builder in the same person-level funnel. Identity continuity across that subdomain boundary is close to nil.

Such a funnel doesn't fail — that's the problem. It returns a precise, confident, catastrophically wrong number, and it looks exactly like every other funnel on the board.

The far side gets joined on referrer or campaign tag instead, and the prohibition is written into the catalogue rather than left as something the team is expected to remember.

The funnel catalogue — every definition ships with the way it can mislead you

Funnel	The question it answers	The caveat attached to it
Assistant handoff opened → first message → trip shape complete → ready to send → sent to a specialist	Of the people who open the chat, how many hand a real trip to a human?	Step 3 is a definition, not a vibe: two or more stops, dates or stated flexibility, passenger count, cabin class. Step 1 was inflated for a fortnight by a scanner burst and is annotated on the chart rather than quietly absorbed — steps 2 onward are unaffected, because a scanner never sends a message.
Main site → builder CTA pageview → engaged-visitor filter → builder CTA clicked	Of people who actually engage with the site, how many click through?	Must never gain a fourth step on the far side of the subdomain boundary. Filtering the denominator to engaged people is what moves this from 2–7% to about 15%; the numerator never changed.
Route page route page viewed → engaged → builder CTA clicked	Do content pages hold attention, and does held attention convert?	Engagement is a disjunction — 25% scrolled or 5 seconds, whichever lands first — so it is generous by design and must never be quoted as a reading-depth metric.
Audience hub hub viewed → audience card clicked	Which audience segments engage, and which of them convert?	The hero CTA is not a card and fires a different event; counting it here would double the apparent card performance. The audience enum had drifted from the live values, splitting some breakdowns silently.
Homepage paths homepage view → chat, direct CTA, or route-builder widget → CTA clicked	Which of the three homepage routes to conversion actually carries traffic?	Three parallel funnels, never summed. The legacy version was eleven steps pinned to the literal text of a button, and it died at step two the moment the button changed. Every step here keys on a typed event instead.
AI-assistant referrals pageview matched to an assistant source → builder CTA clicked	Is traffic arriving from AI assistants, and does it behave differently?	Detection mixes campaign tags and referrer patterns, and one major assistant sends neither, so it is visible only by referrer. Read as a floor, never as a share of total.
Lead-capture lifecycle surface shown → submit attempted → lead created → autoresponse sent → lead activated	Does a captured email actually turn into an engaged lead?	Spans client to server, with the last three steps emitted by the backend so an ad-blocker can shorten the first step but never the outcome. Currently reads zero: the capture surface is behind a flag at zero percent. A defined instrument waiting for traffic, not a result.
Proposal review proposal created → reviewed → approved or rejected	How much of what the AI learning pipeline proposes survives human review?	Specified and deliberately not captured yet. Listed anyway, because a catalogue containing only the funnels that worked is a sales document.

All eight share one convention — ordered steps, other events allowed in between, a 14-day window, counted by unique people — so a reader never has to ask which rules a given funnel is playing by. The third column is the part that usually doesn't exist anywhere.

Screenshot pending

The event contract

One versioned document every capture site codes against: the event, its required and optional properties, the enum each draws from, its PII level, and its double-count guard. The audit that caught the contract breaking its own enum ran against this file.

Screenshot pending

The funnel dashboard

The named funnels as they are actually consumed — conversion steps, drop-off, error and spam rates side by side. A dashboard is only useful if someone can look at it and disagree with you, which means every tile has to say what it excludes.

Where it becomes an AI problem

The team stopped clicking dashboards and started asking

None of the above is machine learning. It becomes an AI problem the moment someone asks the internal assistant “how did lead capture do last week?” — because an agent that can query metrics but not their caveats is a very fast way to industrialise a wrong number.

This is the narrow, defensible version of “the analytics are AI-accessible”. Both tools are read-only and bounded, and the half that actually matters is the second one: the caveats live in the same place the agent retrieves from, which is a precondition for a trustworthy answer rather than a demonstration of one.

The decision worth defending

The instinct is to attach the analytics vendor's hosted MCP server and move on. It was rejected on inspection: its tool-dump mode floods the model's context with schemas, and its exec mode is multi-round-trip discovery against a sixty-second route budget.

There was also a structural reason, and that's the one that decided it. The requirement was the same tools on the chat agent and on the MCP server. An inbound MCP handler can't cleanly proxy an upstream MCP — so adopting the hosted server would have meant the two surfaces exposing different tools. One shared module behind two thin adapters is the only topology where they're literally the same thing.

Why the boring constraints matter

A curated enum before a query language. Ten pre-aggregated metrics return a handful of numbers. Handing a model raw SQL means it has to learn the schema, and it can pull back thousands of rows to reason over.
The escape hatch is bounded anyway. A row limit is appended when the model forgets one, results are capped, and it's deliberately uncached — arbitrary SQL has no reusable cache key.
The cache is a fairness mechanism. Query limits are shared org-wide with the team's own dashboards, so an agent querying freely can starve the humans.
Analytics results are data, not sources. The agent is instructed never to footnote a metric the way it footnotes a retrieved document. A number is not a citation, and blurring the two is how a figure acquires authority it hasn't earned.

The half that makes it worth doing at all: the funnel definitions and their caveats are mirrored into the knowledge base the agent retrieves from, as internal-audience entries. So an answer arrives with its limits attached rather than as a bare figure — and the audience gate was verified against the real retrieval path rather than asserted: none of those entries reachable by the customer-facing agents, all of them by the internal one.

Screenshot pending

The internal agent answering from the analytics tools

A question in plain language, the curated metric tool called with an explicit date range, and the result rendered as a chart rather than a wall of JSON. The caveats the answer needs live in the same knowledge base the agent retrieves from.

The hard parts

Four decisions where the cheap option was available and wrong

Anyone can list tools. What's worth reading is the point where the obvious answer was right there and I can tell you exactly what it would have cost.

The bots could not be filtered, so the metric had to be redefined

What was wrong

From late June 2026 raw visitor counts ran five to ten times inflated by corporate link scanners — Microsoft SafeLinks and its equivalents, following every URL in every email that passes through a company mailbox. They arrive as ordinary desktop Chrome with a search-engine referrer. Every obvious fix was available and every one failed: the analytics vendor's own user-agent bot filter does not catch them, and no client-side signal exists at the moment a pageview fires, so nothing can be dropped at ingestion.

What I did

Stop trying to identify bots and change what the metric counts. Engaged visitors — unique people who fire a page-leave event, which a single-pageview scanner never does. The swap was cheap because that event carries the same host, path, device-type and referring-domain properties, so every existing breakdown survived intact. Applied at query time, so the contaminated history stays in the database and stays readable, annotated rather than deleted.

Why the obvious answer was worse

The clever-looking move was to build a cohort of confirmed scanners and exclude it. I built it. Excluding all of them moved a 10,232-visitor week to 10,077 — a 1.5% dent on a population that was mostly bots, because only a sliver of scanners are identifiable that way. A filter that removes 1.5% of a contaminated population is worse than no filter, because it looks like a fix and you stop asking. A metric bots structurally cannot fake needs no maintenance; a blocklist needs it forever and degrades silently in between.

Comparing two eras with an instrument that only exists in one of them

What was wrong

The rebuilt site tags its outbound links with a campaign source. The old site never did. So a chart of that tag shows a channel rising from nothing — a months-of-work-paying-off story generated entirely by the tag being new. The signal that genuinely spans both eras is the referrer, which said arrivals were down about 17%. That looked like the honest answer, and it was also wrong.

What I did

Check whether the instrument itself was the same in both periods. It was not: arrivals carrying an empty or direct referrer rose from 37% to 51% as browsers tightened referrer policy. That degradation alone accounts for the entire apparent gap. The honest read is no detectable change, and that is what got written down — along with the rule that produced it.

Why the obvious answer was worse

This is the dangerous version, because nothing is broken and no query is wrong. Both numbers are real; the comparison between them is void, and there is no way to see that from the chart — the chart looks perfect. The rule that came out of it is the transferable part: before comparing two eras, prove the signal is continuous across the boundary, not merely present on both sides of it.

A rolling window that quietly ate its own baseline

What was wrong

Every before-and-after comparison ran on a rolling last-N-days window. Reasonable, conventional, and wrong: as the post-launch era aged, the window scrolled the pre-launch baseline out of range. About five weeks after launch it contained zero pre-cutover days and every comparison silently became vacuous. Nothing errored. No query failed. The charts kept rendering.

What I did

Anchor the window to the event rather than to now — a fixed start computed as the cutover date minus 28 days, so the baseline stays put while the post-launch side grows to today. The cutover date became a single exported constant, so every comparison anchors to the same day instead of to whenever someone happened to load the page.

Why the obvious answer was worse

It is the purest example of a metric that fails without failing. There is no alert for a comparison that has become meaningless, no error rate that moves, no test that goes red. The only defence is that the definition is written down somewhere a person can review it against what it was supposed to mean — which is the argument for a definition catalogue that a dashboard alone can never make.

The event contract was breaking its own rule, and only an audit found it

What was wrong

A written contract only helps if it is true. The main conversion event carries a fourteen-value enum for the surface a click came from. Auditing that enum against the document's own instructions to implementers turned up a fifteenth value — named twice in the prose, declared in no property block anywhere in the file. So either the capture site fails typecheck, or the enum is quietly wider than documented and a breakdown is splitting.

What I did

Audit enums against usage, not just against themselves, and bind them to a source of truth with an update obligation. Where an enum mirrors the CMS, the contract now names the CMS as canonical, date-stamps the mirror so staleness is detectable, and requires the update in the same pull request. A parallel case on a second enum was worse: nine specified values against twelve live ones, with three near-misses differing only by a plural.

Why the obvious answer was worse

This is the self-referential version of the whole page. The document whose entire job is preventing drift had drifted, and nothing surfaced it until someone deliberately went looking. A contract nobody audits is a comment.

What this isn't

The limits, said out loud

A boundary you find here costs you ten seconds. A boundary you find three emails into a project costs considerably more.

None of this is a conversion improvement

The move from 2–7% to about 15% is one behaviour measured two ways. Presenting it as a lift would be exactly the failure this page is about, so it is framed as a correction everywhere — including in the diagram, which draws the numerator identically on both sides on purpose.

Traffic did not grow after the rebuild

The apparent jump was scanner bots. Engaged visitors were flat across the cutover. Pages per visitor halving is what gave it away, and that had been visible on the dashboard the whole time.

One capture surface never produced a lead

It shipped complete and correct — instrumented, monitored, idempotent, mounted on every route page — behind a flag at zero percent, at the end of a five-day sprint whose load-bearing deliverables were the measurement and the durable path. The staged rollout was deferred to its own piece of work rather than rushed on the last afternoon.

No year-on-year comparison was possible

Analytics history only begins around July 2025, when the previous site first added the tracking library. Stated up front rather than discovered when someone asked for it.

Nothing here measures whether the AI answers correctly

The tools the agents call are read-only and bounded, and the definitions they read from are audience-gated. But there is no eval harness over the answers, and one metric the agent can still reach is knowingly bot-inflated — flagged do-not-quote in the docs with an open follow-up. That is a convention, not a guardrail, and I would rather say it here than have you find it.

I did not own the whole funnel

The self-serve builder is a different codebase owned by a different engineer, and the CRM is decades old and not mine. The hardest problem here — the identity boundary — exists precisely because the funnel crosses systems I do not control.

Where every number on this page came from

Figure	Kind	Source, window and denominator
46 events pinned by a written contract before the code	counted	46 distinct events and 147 typed property slots, drawn from 84 reused names so the same name means the same thing everywhere. Not documentation of what got built — the spec the implementations had to obey.
8 named funnels, each published with the caveat that limits it	counted	One shared convention across all eight — ordered steps, a 14-day window, counted by unique people — so nobody has to ask which rules a given funnel is playing by.
5–10× traffic inflation from scanner bots, found and neutralised	measured	Raw visitor counts on the main site from late June 2026, against engaged visitors over the same weeks. Corporate link scanners, presenting as ordinary desktop Chrome. Not filterable at ingestion — the fix was a change of definition.
2 read-only analytics tools an LLM agent can actually call	counted	A curated metric tool and a bounded query escape hatch, added to the team's chat agent and to an MCP server from one shared module — so a metric added once appears on both surfaces.
600–960 engaged visitors per week — flat straight across the rebuild	measured	Weekly unique people firing a page-leave event, sixteen consecutive weeks either side of the June 2026 cutover. Raw counts over the same weeks ran five to ten times higher and climbing. The flat line is the control: real demand did not move.
~15% vs 2–7% same-site conversion, correctly denominated versus bot-diluted	measured	One funnel read twice. Identical numerator; the denominator is raw visitors in the first reading and engaged visitors in the second. A correction, not an improvement — nothing about the behaviour changed.
1.2% vs 15.5% desktop versus mobile CTA rate — the tell that the desktop bulk wasn't human	measured	Click rate by device over a fixed four-week baseline window. A thirteen-fold device gap in the same product is not a device gap. It sat on the dashboard for weeks before anyone thought to look, because the headline number was going up.
1.5% how far the obvious bot fix actually moved the number	measured	Excluding an entire cohort of confirmed scanners moved a 10,232-visitor week to 10,077. That failed fix is the evidence that the answer had to be a change of definition rather than a longer exclusion list.

FAQ

What people ask about this work

Including the one that matters most: no, the conversion rate did not improve.

Isn't this just setting up analytics?

Installing the tool is an afternoon. The work is deciding what each event means before the code exists, and then whether a given number is allowed to inform a decision — because a wrong one does not sit quietly. It buys a redesign, kills a channel, or settles an argument it had no business settling.

The output is a written contract every capture site codes against, funnels published with the specific way each one can mislead you, and dashboards that survive being interrogated.

Why not just block the bots?

Tried it, and it is the most useful failed fix in the whole story. Excluding every confirmed scanner moved a 10,232-visitor week to 10,077 — about one and a half percent, on a population that was mostly automated. Only a sliver of them are identifiable that way, and the vendor's own bot filter misses the rest.

The answer had to be a change of definition rather than a longer exclusion list: count people who fire an event a single-pageview scanner structurally cannot fire. Applied at query time, so the contaminated history stays readable and annotated rather than deleted.

So did the conversion rate actually improve?

No, and presenting it as an improvement would be exactly the failure this page is about. The move from two-to-seven percent to about fifteen is one behaviour measured two ways — identical numerator, an honest denominator. Nothing about the site got better.

Traffic did not grow after the rebuild either. Engaged visitors were flat straight across the cutover; the apparent jump was scanner traffic.

We already have a year of dirty data. Is it salvageable?

Usually, yes, and you should not delete it. Contamination is fixable at query time if you can find something the contaminating traffic structurally cannot fake — the history stays in the database, annotated rather than dropped, so old numbers remain reproducible and their correction is visible.

What is not salvageable is a period where nobody wrote down what an event meant. That is the part worth fixing before the next launch rather than after it.

What's the first thing you'd produce for us?

The event contract, before any code changes. Every event, its required and optional properties, the enum each draws from, a PII level and a double-count guard — one versioned document that the implementations have to obey rather than a description of whatever got built.

It is also the cheapest artefact to argue with. Most of the disagreements worth having about measurement surface while writing that document, which is much better than surfacing them in a dashboard six months later.

Next step

If you're about to make a decision on a number nobody has audited

This transfers cleanly, and it's usually cheaper than the decision it prevents. Same pattern wherever it lands: find what the metric is actually counting, define it so it can't be faked, and write the caveat next to the definition rather than in someone's head.

Ask me to audit a funnel Read the systems write-up

The closed-loop write-up covers the four systems this measurement layer sits under — the chat agent, the reconciliation backend and the knowledge platform.