NC Nati Correa
AI systems and agents
40.7128 N / 74.0060 W
LOG / 07
/ Case study / Method / Published numbers

The lean test

The A/B test behind my /lean skill: 24 overnight runs, graded blind, and the number that changed how I use it.
/ Why lean exists

Everything Claude builds while I sleep comes out of the same budget I need during the day, and waste was quietly eating it. So I wrote lean: one file of rules Claude reads before starting any big job. I was sure it was saving me money, but sure isn't proof, so I built a test.

/ What the rules say

It requires the plan to get written and every question to get asked before work starts, so mid-build assumptions never happen. It delegates bulk reading and grunt work to smaller, cheaper models, so the expensive one spends its tokens doing the actual reasoning. Since output costs about 5x what input does, it edits with small diffs and never pastes whole files back into the chat. It verifies proportionally, testing exactly what got changed, with no claiming done without proof. And it protects session hygiene: anytime a session hits 45% context, it wraps up and restarts fresh so quality never degrades.

24runs, one night
26%cheaper on the heavy build
16/16best artifact, a lean run
±1quality band, every task
/ The test

I took the same 4 real tasks from an app I'm building and ran each one 3 times with lean and 3 times without, 24 runs total overnight. The repo was frozen at one commit and every run got its own isolated copy, so all 24 built the same thing without touching each other. A separate Claude graded every output blind against a checklist I wrote before the test, without ever knowing which runs used the skill.

/ What came back

The big build came out 26% cheaper. Quality stayed within one point everywhere, one task actually scored higher with lean on, and the single best output of all 24 runs came from a lean run. These comparisons used the same exact prompt, and the only difference was lean.

The catch is that lean costs more than it saves on quick little tasks, because reading the rulebook is itself tokens. My reading task ran 41% more expensive with the skill on, so now lean only gets invoked on big work. And the 16/16 dashboard the test produced was good enough that it shipped as the app's real dashboard.

24 runs is a small test: 3 per cell, one model, my tasks, and the 26% is a median with real variance behind it. Both arms also ran with my usual base config, which already carries some token discipline, so if anything the test understates what lean does on a bare setup.

/ Proof
Medians of 3, token units weighted input/cache/outputT1 HEAVY BUILD .......... 26% CHEAPER (0.784 VS 1.059 UNITS)
T2 SMALL BUILD .......... WASH (4%)
T3 MEDIUM CODE .......... 11% PRICIER
T4 PURE READING ......... 41% PRICIER
QUALITY ................. ALL MEDIANS WITHIN 1 POINT
BEST ARTIFACT ........... 16/16 DASHBOARD, FROM A LEAN RUN
VERDICT ................. INVOKE ON BIG WORK, SKIP ON SMALL
The lean skill, free in the public repo
Back to the build log