My role
Owner, designer and sole builder
Team
My team lead and two onboarding teammates as users and reviewers
Timeline
June to October 2026
Outcome
An AI assistant that researches new accounts and preps the onboarding call, public in PostHog’s skills repo, 41% faster after a rebuild and running in PostHog’s Slack bot
1,448
new accounts entered our onboarding queue from January to August, each researched by hand
41%
less agent time for the same research after I rebuilt it
80k → 40k
words of instructions, cut in half before it went public
Why it mattered
Before PostHog’s onboarding team reaches out to a new customer, someone has to understand the account: what they pay for, how they’ve set up each product, and whether a billing limit is quietly throwing away their data. All of that was done by hand, across a SQL tool and our CRM, while the median account waited 9 days for a first outreach and 12.9% were never contacted at all.
I’d already built an AI research agent for myself on Claude Code that did this work. In June I demoed it to my team lead, Magda, and pitched one shared assistant for the whole team. She asked me to own it, and it became my public Q3 goal: research and call prep for every new onboarding account, so the team could spend more time on the accounts with the most potential.
The bar was trust, because you can’t go back for a figure in the middle of a customer call.
What I built
First I made it work for anyone. Each teammate runs it with their own keys against separate US and EU data, so I split it into a short core file plus reference files and stripped out everything specific to me. Side by side with my old version on three real accounts, the new one won on correctness after 12 fixes.
Tests in place of eyeballing
Teammates found defects I’d missed, and a 10x arithmetic error reads perfectly fine to a human. So in July I built an evaluation suite: ground truth captured live from one US and one EU account, plus independent grader agents that score every run against it. Now every change gets tested before it ships.
Faster by batching the work
On two accounts the old version made 473 tool calls, every agent working one call at a time, so a run took as long as the longest chain inside a single agent. More agents would have spent capacity the whole team shares, while batching costs nothing. So I rebuilt it around batching, with a rule that the speed budget shapes the work and never cuts what gets read.
Smaller where it counts
In August I cut the instructions from about 80,000 words to 40,000. A review panel across two AI model families then showed the remaining bloat sat in the core file, the only one loaded on every run, so that part lost another 21.3%.
A person reads everything
The assistant never sends, posts or writes anything, and every number has to come from a query it ran in that run. On a hard customer question it’s, as I told a new teammate, “the skill solving it 90% of the way and then I just edit”.
What happened
It merged into PostHog’s public skills repo on 17 August, closing the shared-tool half of my Q3 goal. In October I put a lighter copy into PostHog’s own AI, so anyone on the team can run it from the Slack bot without installing anything.
The rebuild cut agent time by 41%. The worst role improved only 2%, because it spent four retries fighting a query instruction that was simply wrong. The new version also found five things the old one missed and lost one, which I fixed by moving the rule from my prompt into the assistant itself.
Magda found the limits. On a simple account, a single notebook prompt got her to the same conclusions, so in her view the assistant earns its time on complex accounts. One of my own call preps that week took 55 minutes, about three times what I’d need by hand, and she called another brief very good but verbose.
The other half of the goal didn’t happen. Moving the team’s time to high-potential accounts depended on a customer segment another team owned, and that segment was never defined.
Whatever we can optimize and implement for our own… selfish goals, it’s actually working for the whole.
Magda Olszewska, onboarding team lead, when the skill went into Slack
What I’d do differently
I’d build the evaluation suite before the first shared version. My teammates found the defects that convinced me, and that’s backwards for a tool people trust on calls.
I’d also scale each run to the account, so a simple account gets a light pass and a complex one gets the full research, and I’d track how often teammates run it from day one, so adoption is a number I can show.
