Turning my personal AI research agent into a tool my whole team runs

Why this mattered

Every account in PostHog’s onboarding queue needs someone to understand it before we reach out: what they pay for, how they set up each product, and whether a billing limit is quietly throwing away their data. Between January and August, 1,448 accounts entered that queue, and the research was done by hand, account by account, across a SQL tool and the CRM. The median account waited 9 days for our first outreach, and 12.9% were never contacted at all.

I’d spent a couple of months building a Claude Code skill for myself that did that research, with a library of SQL queries and parallel sub-agents that each read one part of the account. On 24 June I demoed it to my team lead, Magda, and pitched one shared skill so nobody on the team does this research by hand anymore. She asked me to own it, and on 1 July it became my public Q3 goal: research and call prep for every new onboarding account, so the team could spend more of its time on high-potential accounts.

So the job was to take a tool that worked for me and make it something my teammates would trust on a live customer call.

Constraints

I built it alongside my own onboarding calls, so every hour on the skill came out of the queue it was meant to help.

Each teammate runs it on their own machine with their own keys, and US and EU are separate instances, so my file paths, shortcuts and secrets all had to go. The data underneath kept moving too, because the CRM we read from was being switched off in October and its replacement was still being built.

The real bar was trust. A wrong number on a customer call costs more than the research time it saved, because you can’t go back for a figure mid-call.

What I did

30 June, the first shared version

I split my personal skill into a short core file plus reference files, stripped out everything specific to me and added a per-user config. Then I ran old and new side by side on three real accounts, and after 12 verified fixes the new one won on correctness and completeness.

17 July, an evaluation suite

Once teammates were running it, defects showed up that I hadn’t seen. So I built fixtures with ground truth captured live from two real accounts, one US and one EU, plus independent grader agents that scored each run against it. Every test passed after five iterations.

12 August, speed

On two accounts the old version used 473 tool calls and about 156 minutes of agent compute, and every one of its twelve gathering agents ran fully serially at 20 to 31 seconds per call. That meant wall clock followed the longest chain of calls inside one agent, however many agents there were. I rebuilt it around batching and had four separate agents verify the thirteen fixes, including a fresh agent following the instructions cold.

13 August, size

I cut the skill from about 80,000 words to 40,000 and moved task management into its own skill.

17 August, review and ship

I ran a cross-model review panel over it, four Cursor models plus five Codex tasks, and had two live account runs audit themselves against a checklist. Both failed the same three checks, which became ten fixes. The skill merged into PostHog’s public skills repo the same day, and a second PR added the install steps the first one missed.

26 October, Slack

I published a lighter copy into PostHog’s own AI, so the team can run it from the Slack bot without installing anything. Every teammate is an owner, and a daily job keeps it in step with the public repo.

The calls I made

Graders over eyeballing

Until July I read outputs and fixed what looked wrong, but a 10x arithmetic error reads perfectly fine. The suite caught exactly that, along with a dropped EU guardrail and a query batch that silently trimmed its own list. It took time away from the queue, and in exchange every future change gets tested before it ships.

Batch, don’t split

The obvious fix for a slow agent is more agents. I didn’t take it, because splitting spends concurrency that everyone running the skill shares, and batching costs nothing. The risk with any speed target is that an agent hits it by reading less, so I wrote a rule that the budget caps the shape of the work, never its scope. On the first real run one agent hit a rate limit on the EU API, slowed down and logged the overrun instead of dropping reads.

Cut where it loads

The panel showed the query files were dense on purpose, and the bloat was in the core file, the only one loaded on every run. So instead of trimming every file evenly, I cut the always-loaded set by 21.3% and took the reference files from 13 to 10. I also fixed five contradictions I’d introduced myself the day before, including a speed optimization that had silently broken the guard for customers with multiple organizations.

A human reads everything

The skill never sends, posts or writes anything. Every number has to come from a query it ran in that run, every product claim from a docs search in that run, and prices or settings are never written into the skill. On a hard customer question it’s, as I told a new teammate, “the skill solving it 90% of the way and then I just edit”.

What happened

The shared-skill half of my Q3 goal was done on 17 August.

Across the 12 roles that exist in both versions, total agent time went from 8,356 seconds to 4,906, 41% faster, and tool calls from 375 to 244. The best role was 56% faster and the worst only 2%, because it spent four retries fighting a query recipe in the skill that was simply wrong.

Quality moved both ways. The new version found five things the old one missed and lost one, flagging paid invoices as unpaid, because the rule that prevents that lived in my prompt instead of the skill. I moved it into the skill.

Magda found the limits. On a thin account, a single notebook prompt got her to the same conclusions, so in her view the skill earns its time on complex accounts and not light ones. That same week a call prep of mine took 55 minutes, and I said myself I could finish my own prep about three times in that time. On another call brief she said it was very good but verbose.

In Slack, a finished call prep got lost because Slack only posts the reply that ends a run’s first turn, so the skill now finishes in one turn. The Slack copy also can’t reach every source the desktop version can.

The other half of the goal didn’t happen. The shift to high-potential accounts depended on a segment another team owned, and that segment was never defined.

Whatever we can optimize and implement for our own… selfish goals, it’s actually working for the whole.

Magda Olszewska, onboarding team lead, when the skill went into Slack

What I’d do differently

I’d build the evaluation suite before the first shared version, instead of three weeks after it. My teammates found the defects that convinced me, and that’s backwards for a tool people trust on calls.

I’d design for the light account from the start. Magda saw before I did that a run that checks everything twice is too heavy for a simple account, so the depth of a run should follow how complex the account is.

I’d write the brief for the person on the call, because verbose is a real problem when they have a customer waiting.

And I’d track how often teammates run it from day one, so adoption is a number I can show rather than something I hear about in meetings.