← All work

PostHog · 2026

Building the AI research assistant behind PostHog’s onboarding calls

I turned my personal AI research agent into an assistant PostHog’s onboarding team runs before customer calls. Grader agents test it, a rebuild made it 41% faster, and it’s public on GitHub and in our Slack bot.

Drawing of a checklist with two green checks and a red cross under a magnifying glass, labelled Graded
My role

Owner, designer and sole builder

Team

My team lead and two onboarding teammates as users and reviewers

Timeline

June to October 2026

Outcome

An AI assistant that researches new accounts and preps the onboarding call, public in PostHog’s skills repo, 41% faster after a rebuild and running in PostHog’s Slack bot

1,448

new accounts entered our onboarding queue from January to August, each researched by hand

41%

less agent time for the same research after I rebuilt it

80k → 40k

words of instructions, cut in half before it went public

Why it mattered

Before PostHog’s onboarding team reaches out to a new customer, someone has to understand the account: what they pay for, how they’ve set up each product, and whether a billing limit is quietly throwing away their data. All of that was done by hand, across a SQL tool and our CRM, while the median account waited 9 days for a first outreach and 12.9% were never contacted at all.

I’d already built an AI research agent for myself on Claude Code that did this work. In June I demoed it to my team lead, Magda, and pitched one shared assistant for the whole team. She asked me to own it, and it became my public Q3 goal: research and call prep for every new onboarding account, so the team could spend more time on the accounts with the most potential.

The bar was trust, because you can’t go back for a figure in the middle of a customer call.

What I built

First I made it work for anyone. Each teammate runs it with their own keys against separate US and EU data, so I split it into a short core file plus reference files and stripped out everything specific to me. Side by side with my old version on three real accounts, the new one won on correctness after 12 fixes.

How the research assistant works: a new account goes to seven parallel research agents, their findings are cross-checked against live docs, and a person gets a call briefINPUTRESEARCH AGENTSVERIFYOUTPUTNew accountname or emailBilling and spendUsage over timeApp engagementEvent mixIdentity and SDKsProduct setupWebsite scanCross-check thefindings andverify claimsin live docsCall brieffor a personEvery number comes from a query in that run. A person reads every brief.
Each research agent reads one part of the account in parallel, and every claim is checked before the brief reaches a person.

Tests in place of eyeballing

Teammates found defects I’d missed, and a 10x arithmetic error reads perfectly fine to a human. So in July I built an evaluation suite: ground truth captured live from one US and one EU account, plus independent grader agents that score every run against it. Now every change gets tested before it ships.

The evaluation loop: fixtures with live ground truth, a full run, grader agents scoring it, fixes and reruns until every test passesRUN BEFORE EVERY CHANGE SHIPSFixtures2 real accountsground truthcaptured liveRunthe assistantend to endGradeindependentgrader agentsscore the runShipevery testpassed after5 roundsFail: fix the skill and rerunWHAT THE GRADERS CAUGHTA 10x arithmetic errorA dropped EU guardrailA silently trimmed list
Grader agents score every run against ground truth from real accounts, and anything that fails gets fixed and rerun.

Faster by batching the work

On two accounts the old version made 473 tool calls, every agent working one call at a time, so a run took as long as the longest chain inside a single agent. More agents would have spent capacity the whole team shares, while batching costs nothing. So I rebuilt it around batching, with a rule that the speed budget shapes the work and never cuts what gets read.

Smaller where it counts

In August I cut the instructions from about 80,000 words to 40,000. A review panel across two AI model families then showed the remaining bloat sat in the core file, the only one loaded on every run, so that part lost another 21.3%.

A person reads everything

The assistant never sends, posts or writes anything, and every number has to come from a query it ran in that run. On a hard customer question it’s, as I told a new teammate, “the skill solving it 90% of the way and then I just edit”.

What happened

It merged into PostHog’s public skills repo on 17 August, closing the shared-tool half of my Q3 goal. In October I put a lighter copy into PostHog’s own AI, so anyone on the team can run it from the Slack bot without installing anything.

The rebuild cut agent time by 41%. The worst role improved only 2%, because it spent four retries fighting a query instruction that was simply wrong. The new version also found five things the old one missed and lost one, which I fixed by moving the rule from my prompt into the assistant itself.

Before and after the rebuild: agent time 8,356 to 4,906 seconds, tool calls 375 to 244, instructions about 80,000 to 40,000 wordsAGENT TIME, 12 ROLES41% fasterBefore8,356 sAfter4,906 sTOOL CALLS, 12 ROLES35% fewerBefore375After244WORDS OF INSTRUCTIONSHalf the sizeBefore~80,000After~40,000
The August rebuild measured on the same two accounts, plus the size of the instructions after the cut the next day.

Magda found the limits. On a simple account, a single notebook prompt got her to the same conclusions, so in her view the assistant earns its time on complex accounts. One of my own call preps that week took 55 minutes, about three times what I’d need by hand, and she called another brief very good but verbose.

The other half of the goal didn’t happen. Moving the team’s time to high-potential accounts depended on a customer segment another team owned, and that segment was never defined.

Whatever we can optimize and implement for our own… selfish goals, it’s actually working for the whole.

Magda Olszewska, onboarding team lead, when the skill went into Slack

What I’d do differently

I’d build the evaluation suite before the first shared version. My teammates found the defects that convinced me, and that’s backwards for a tool people trust on calls.

I’d also scale each run to the account, so a simple account gets a light pass and a complex one gets the full research, and I’d track how often teammates run it from day one, so adoption is a number I can show.