# Analytics & measurement > The instrument — events pinned before the code, eight funnels rebuilt, and the finding that most of the recorded traffic was never human. - Canonical: https://iambogdan.com/case-studies/analytics - Author: Bogdan Andronic (AI Engineer & Full-Stack TypeScript Developer) - Source: generated from the data modules this page renders - Records: 32 - Part of: A US travel company (Built for a client · anonymised) ## Measured ### events pinned by a written contract before the code 46 distinct events and 147 typed property slots, drawn from 84 reused names so the same name means the same thing everywhere. Not documentation of what got built — the spec the implementations had to obey. - value: 46 - kind: counted ### named funnels, each published with the caveat that limits it One shared convention across all eight — ordered steps, a 14-day window, counted by unique people — so nobody has to ask which rules a given funnel is playing by. - value: 8 - kind: counted ### traffic inflation from scanner bots, found and neutralised Raw visitor counts on the main site from late June 2026, against engaged visitors over the same weeks. Corporate link scanners, presenting as ordinary desktop Chrome. Not filterable at ingestion — the fix was a change of definition. - value: 5–10× - kind: measured ### read-only analytics tools an LLM agent can actually call A curated metric tool and a bounded query escape hatch, added to the team's chat agent and to an MCP server from one shared module — so a metric added once appears on both surfaces. - value: 2 - kind: counted ### engaged visitors per week — flat straight across the rebuild Weekly unique people firing a page-leave event, sixteen consecutive weeks either side of the June 2026 cutover. Raw counts over the same weeks ran five to ten times higher and climbing. The flat line is the control: real demand did not move. - value: 600–960 - kind: measured ### same-site conversion, correctly denominated versus bot-diluted One funnel read twice. Identical numerator; the denominator is raw visitors in the first reading and engaged visitors in the second. A correction, not an improvement — nothing about the behaviour changed. - value: ~15% vs 2–7% - kind: measured ### desktop versus mobile CTA rate — the tell that the desktop bulk wasn't human Click rate by device over a fixed four-week baseline window. A thirteen-fold device gap in the same product is not a device gap. It sat on the dashboard for weeks before anyone thought to look, because the headline number was going up. - value: 1.2% vs 15.5% - kind: measured ### how far the obvious bot fix actually moved the number Excluding an entire cohort of confirmed scanners moved a 10,232-visitor week to 10,077. That failed fix is the evidence that the answer had to be a change of definition rather than a longer exclusion list. - value: 1.5% - kind: measured ## Funnels ### Assistant handoff Of the people who open the chat, how many hand a real trip to a human? - steps: opened → first message → trip shape complete → ready to send → sent to a specialist - caveat: Step 3 is a definition, not a vibe: two or more stops, dates or stated flexibility, passenger count, cabin class. Step 1 was inflated for a fortnight by a scanner burst and is annotated on the chart rather than quietly absorbed — steps 2 onward are unaffected, because a scanner never sends a message. ### Main site → builder CTA Of people who actually engage with the site, how many click through? - steps: pageview → engaged-visitor filter → builder CTA clicked - caveat: Must never gain a fourth step on the far side of the subdomain boundary. Filtering the denominator to engaged people is what moves this from 2–7% to about 15%; the numerator never changed. ### Route page Do content pages hold attention, and does held attention convert? - steps: route page viewed → engaged → builder CTA clicked - caveat: Engagement is a disjunction — 25% scrolled or 5 seconds, whichever lands first — so it is generous by design and must never be quoted as a reading-depth metric. ### Audience hub Which audience segments engage, and which of them convert? - steps: hub viewed → audience card clicked - caveat: The hero CTA is not a card and fires a different event; counting it here would double the apparent card performance. The audience enum had drifted from the live values, splitting some breakdowns silently. ### Homepage paths Which of the three homepage routes to conversion actually carries traffic? - steps: homepage view → chat, direct CTA, or route-builder widget → CTA clicked - caveat: Three parallel funnels, never summed. The legacy version was eleven steps pinned to the literal text of a button, and it died at step two the moment the button changed. Every step here keys on a typed event instead. ### AI-assistant referrals Is traffic arriving from AI assistants, and does it behave differently? - steps: pageview matched to an assistant source → builder CTA clicked - caveat: Detection mixes campaign tags and referrer patterns, and one major assistant sends neither, so it is visible only by referrer. Read as a floor, never as a share of total. ### Lead-capture lifecycle Does a captured email actually turn into an engaged lead? - steps: surface shown → submit attempted → lead created → autoresponse sent → lead activated - caveat: Spans client to server, with the last three steps emitted by the backend so an ad-blocker can shorten the first step but never the outcome. Currently reads zero: the capture surface is behind a flag at zero percent. A defined instrument waiting for traffic, not a result. ### Proposal review How much of what the AI learning pipeline proposes survives human review? - steps: proposal created → reviewed → approved or rejected - caveat: Specified and deliberately not captured yet. Listed anyway, because a catalogue containing only the funnels that worked is a sales document. ## Projects ### Product analytics & measurement layer 46 events and 147 typed property slots pinned in a contract before the code, with double-count guards and a PII level per event. The finding that justified the whole layer: raw visitor counts were running five to ten times inflated by corporate link scanners, which cannot be dropped at ingestion and which the vendor's own bot filter misses — so the metric was redefined to something bots structurally cannot fake rather than maintained as a blocklist. Same-site conversion read 2–7% on the wrong denominator and about 15% on the right one. The definitions and their caveats are mirrored into the knowledge platform, and two read-only query tools put them behind both an LLM agent and an MCP server from one shared module. - kind: Production · analytics - summary: Their visitor numbers were running five to ten times too high — corporate link scanners the vendor's own bot filter never caught. I rewrote what every event means, rebuilt eight funnels, and got conversion measured against a denominator that's actually true. - stack: PostHog · HogQL · Next.js 16 · Event contract design · MCP · Mastra - status: live ## The hard parts ### The bots could not be filtered, so the metric had to be redefined From late June 2026 raw visitor counts ran five to ten times inflated by corporate link scanners — Microsoft SafeLinks and its equivalents, following every URL in every email that passes through a company mailbox. They arrive as ordinary desktop Chrome with a search-engine referrer. Every obvious fix was available and every one failed: the analytics vendor's own user-agent bot filter does not catch them, and no client-side signal exists at the moment a pageview fires, so nothing can be dropped at ingestion. - fix: Stop trying to identify bots and change what the metric counts. Engaged visitors — unique people who fire a page-leave event, which a single-pageview scanner never does. The swap was cheap because that event carries the same host, path, device-type and referring-domain properties, so every existing breakdown survived intact. Applied at query time, so the contaminated history stays in the database and stays readable, annotated rather than deleted. - why: The clever-looking move was to build a cohort of confirmed scanners and exclude it. I built it. Excluding all of them moved a 10,232-visitor week to 10,077 — a 1.5% dent on a population that was mostly bots, because only a sliver of scanners are identifiable that way. A filter that removes 1.5% of a contaminated population is worse than no filter, because it looks like a fix and you stop asking. A metric bots structurally cannot fake needs no maintenance; a blocklist needs it forever and degrades silently in between. ### Comparing two eras with an instrument that only exists in one of them The rebuilt site tags its outbound links with a campaign source. The old site never did. So a chart of that tag shows a channel rising from nothing — a months-of-work-paying-off story generated entirely by the tag being new. The signal that genuinely spans both eras is the referrer, which said arrivals were down about 17%. That looked like the honest answer, and it was also wrong. - fix: Check whether the instrument itself was the same in both periods. It was not: arrivals carrying an empty or direct referrer rose from 37% to 51% as browsers tightened referrer policy. That degradation alone accounts for the entire apparent gap. The honest read is no detectable change, and that is what got written down — along with the rule that produced it. - why: This is the dangerous version, because nothing is broken and no query is wrong. Both numbers are real; the comparison between them is void, and there is no way to see that from the chart — the chart looks perfect. The rule that came out of it is the transferable part: before comparing two eras, prove the signal is continuous across the boundary, not merely present on both sides of it. ### A rolling window that quietly ate its own baseline Every before-and-after comparison ran on a rolling last-N-days window. Reasonable, conventional, and wrong: as the post-launch era aged, the window scrolled the pre-launch baseline out of range. About five weeks after launch it contained zero pre-cutover days and every comparison silently became vacuous. Nothing errored. No query failed. The charts kept rendering. - fix: Anchor the window to the event rather than to now — a fixed start computed as the cutover date minus 28 days, so the baseline stays put while the post-launch side grows to today. The cutover date became a single exported constant, so every comparison anchors to the same day instead of to whenever someone happened to load the page. - why: It is the purest example of a metric that fails without failing. There is no alert for a comparison that has become meaningless, no error rate that moves, no test that goes red. The only defence is that the definition is written down somewhere a person can review it against what it was supposed to mean — which is the argument for a definition catalogue that a dashboard alone can never make. ### The event contract was breaking its own rule, and only an audit found it A written contract only helps if it is true. The main conversion event carries a fourteen-value enum for the surface a click came from. Auditing that enum against the document's own instructions to implementers turned up a fifteenth value — named twice in the prose, declared in no property block anywhere in the file. So either the capture site fails typecheck, or the enum is quietly wider than documented and a breakdown is splitting. - fix: Audit enums against usage, not just against themselves, and bind them to a source of truth with an update obligation. Where an enum mirrors the CMS, the contract now names the CMS as canonical, date-stamps the mirror so staleness is detectable, and requires the update in the same pull request. A parallel case on a second enum was worse: nine specified values against twelve live ones, with three near-misses differing only by a plural. - why: This is the self-referential version of the whole page. The document whose entire job is preventing drift had drifted, and nothing surfaced it until someone deliberately went looking. A contract nobody audits is a comment. ## What this does not claim ### None of this is a conversion improvement The move from 2–7% to about 15% is one behaviour measured two ways. Presenting it as a lift would be exactly the failure this page is about, so it is framed as a correction everywhere — including in the diagram, which draws the numerator identically on both sides on purpose. ### Traffic did not grow after the rebuild The apparent jump was scanner bots. Engaged visitors were flat across the cutover. Pages per visitor halving is what gave it away, and that had been visible on the dashboard the whole time. ### One capture surface never produced a lead It shipped complete and correct — instrumented, monitored, idempotent, mounted on every route page — behind a flag at zero percent, at the end of a five-day sprint whose load-bearing deliverables were the measurement and the durable path. The staged rollout was deferred to its own piece of work rather than rushed on the last afternoon. ### No year-on-year comparison was possible Analytics history only begins around July 2025, when the previous site first added the tracking library. Stated up front rather than discovered when someone asked for it. ### Nothing here measures whether the AI answers correctly The tools the agents call are read-only and bounded, and the definitions they read from are audience-gated. But there is no eval harness over the answers, and one metric the agent can still reach is knowingly bot-inflated — flagged do-not-quote in the docs with an open follow-up. That is a convention, not a guardrail, and I would rather say it here than have you find it. ### I did not own the whole funnel The self-serve builder is a different codebase owned by a different engineer, and the CRM is decades old and not mine. The hardest problem here — the identity boundary — exists precisely because the funnel crosses systems I do not control. ## Questions and answers ### Isn't this just setting up analytics? Installing the tool is an afternoon. The work is deciding what each event means before the code exists, and then whether a given number is allowed to inform a decision — because a wrong one does not sit quietly. It buys a redesign, kills a channel, or settles an argument it had no business settling. The output is a written contract every capture site codes against, funnels published with the specific way each one can mislead you, and dashboards that survive being interrogated. ### Why not just block the bots? Tried it, and it is the most useful failed fix in the whole story. Excluding every confirmed scanner moved a 10,232-visitor week to 10,077 — about one and a half percent, on a population that was mostly automated. Only a sliver of them are identifiable that way, and the vendor's own bot filter misses the rest. The answer had to be a change of definition rather than a longer exclusion list: count people who fire an event a single-pageview scanner structurally cannot fire. Applied at query time, so the contaminated history stays readable and annotated rather than deleted. ### So did the conversion rate actually improve? No, and presenting it as an improvement would be exactly the failure this page is about. The move from two-to-seven percent to about fifteen is one behaviour measured two ways — identical numerator, an honest denominator. Nothing about the site got better. Traffic did not grow after the rebuild either. Engaged visitors were flat straight across the cutover; the apparent jump was scanner traffic. ### We already have a year of dirty data. Is it salvageable? Usually, yes, and you should not delete it. Contamination is fixable at query time if you can find something the contaminating traffic structurally cannot fake — the history stays in the database, annotated rather than dropped, so old numbers remain reproducible and their correction is visible. What is not salvageable is a period where nobody wrote down what an event meant. That is the part worth fixing before the next launch rather than after it. ### What's the first thing you'd produce for us? The event contract, before any code changes. Every event, its required and optional properties, the enum each draws from, a PII level and a double-count guard — one versioned document that the implementations have to obey rather than a description of whatever got built. It is also the cheapest artefact to argue with. Most of the disagreements worth having about measurement surface while writing that document, which is much better than surfacing them in a dashboard six months later.