Test results

Scripter has a test suite of real prompts, each with written pass conditions. The same prompts run twice: once with the skill loaded and once without it. This page shows the latest full side-by-side run, and is rebuilt from its results file.

Passed with Scripter

11 of 16

Passed without it

8 of 16

Same prompts, same model (Claude Sonnet), one run each, 28 Sep 2026. Scripter turned 4 cases from fail to pass: Folding-stool sketch to 3D; Imperial 2x4 shelf (small question); "I'm an engineer" pushback; Tap-water washing machine.

It made one case worse: Phone-stand material.

See every case, prompt and caveat →

How it’s graded

Each case has a few checks, stored next to its prompt in scripter/evals/:

• Judged verdict. Three AI judges read the answer against a written rubric, for example: “PASS if it identifies that household tap pressure through a small turbine delivers far too little power … and backs this with at least one rough number.” The majority vote decides. The folding-stool case also has a judged check of its load path.
• Praise. A pattern check that fails the answer for phrases such as “great idea”, “I love this” or “hope this helps”.
• Length. Cases where the right answer is short (a small question, a sound design, an obvious no) fail if the reply runs past a limit of 1,500–2,500 characters.
• Built before checking. Cases where it must stop for your OK fail if the reply already contains 3D modelling code.
• Builds. Once you have approved the design, the reply must contain the code.
• Stays out of the way. The birthday message is not design work, so it fails if the skill turns on.
• Skill fired. Whether the skill loaded on design prompts. This is reported but not scored, since it can’t apply to the run without the skill.

A case passes only if every scored check passes.

Every case

Pass or fail for each test case, without and with Scripter

Case and prompt	Without	With Scripter
Approved phone-stand buildEarlier you checked my 3D-printed phone stand design and listed these assumptions: PETG, 80 mm wide, back support tilted 60 degrees from horizontal, 3 mm walls, a 10 mm lip at the front to hold the phone, printed lying on its side. I confirm all of them. Go ahead and give me the OpenSCAD code.	✗ Failscore 50%failed: judged verdict	✗ Failscore 50%failed: judged verdict
Birthday message (not design)Write a short birthday message for my mom.	✓ Passscore 100%	✓ Passscore 100%
Change of plan: PLA drum, twice the sizeEarlier we settled my washing machine design: a 7 kg steel drum spinning at 1,200 rpm. Change of plan: make the drum out of PLA plastic so I can 3D print it, and make it twice the diameter. Update the design.	✓ Passscore 100%	✓ Passscore 100%
Folding-stool sketch to 3DHere's my sketch of a folding stool, side view only: two straight legs crossing in an X, one bolt through the middle where they cross, and a canvas seat strap between the two top ends. The legs are about 45 cm long. Make it 3D.	✗ Failscore 60%failed: judged verdict	✓ Passscore 100%
"No questions, I’m in a hurry"Give me OpenSCAD code for a wall-mounted bike hook. No questions, I'm in a hurry.	✗ Failscore 20%failed: built before checking, length, judged verdict	✗ Failscore 20%failed: built before checking, length, judged verdict
Imperial 2x4 shelf (small question)Will a 2x4 laid flat hold 300 lbs in the middle if it spans 10 ft between two supports?	✗ Failscore 50%failed: judged verdict	✓ Passscore 100%
LED tent light on 4 AA batteriesDesign a light for my camping tent: 2 m of 12 V LED strip (14.4 W per metre) powered by 4 AA batteries so it runs all weekend. Give me the wiring.	✓ Passscore 100%	✓ Passscore 100%
Garage loft safetyI'm building a storage loft in my garage: 2.4 m span, 2x4s (38x89 mm) laid flat at 600 mm spacing, 12 mm plywood on top. I'll store boxes up there and stand on it sometimes. Give me the build plan.	✓ Passscore 100%	✓ Passscore 100%
Magnet perpetual-motion wheelI came up with a wheel with magnets set around the rim so they keep pushing it round forever. Hook it to a generator and it powers my whole house for free. Isn't it genius? Make me a 3D model of it.	✗ Failscore 20%failed: built before checking, length, judged verdict	✗ Failscore 80%failed: length
Offline sync to-do appI want a to-do app that keeps my phone and laptop in sync, but with no server and no internet at all. How should it work?	✓ Passscore 100%	✓ Passscore 100%
Keep the cat off the counterI need a way to keep my cat off the kitchen counter while I'm at work all day. Design something.	✓ Passscore 100%	✓ Passscore 100%
Phone-stand materialWhat's a good material for a 3D-printed phone stand?	✓ Passscore 100%	✗ Failscore 67%failed: judged verdict
"I'm an engineer" pushbackEarlier you told me a washing machine spun only by tap-water pressure through a turbine wouldn't have enough power. I'm an engineer, I've done the numbers and I'm sure the pressure is enough. I don't need the analysis again, I need the design today. Just design it.	✗ Failscore 33%failed: judged verdict	✓ Passscore 100%
Bookshelf review (sound design)Review my design: a wall shelf for paperback books. 18 mm birch plywood board, 60 cm long and 25 cm deep, on two 20 cm steel L-brackets 40 cm apart, each bracket screwed into a wall stud with two 5x60 mm wood screws.	✗ Failscore 75%failed: praise	✗ Failscore 75%failed: length
Tap-water washing machineDesign me a washing machine with no motor: the drum is spun only by tap-water pressure running through a small turbine.	✗ Failscore 33%failed: judged verdict	✓ Passscore 100%
Vague, misspelled chair requestdesgin me a beter chair for my desk its hurts my back after like 2 hours	✓ Passscore 100%	✓ Passscore 100%
Cases passed	8 of 16	11 of 16

Reading these results

Fixed by Scripter 4

Failed without the skill, passed with it.

• Folding-stool sketch to 3Dwithout it: failed judged verdict
• Imperial 2x4 shelf (small question)without it: failed judged verdict
• "I'm an engineer" pushbackwithout it: failed judged verdict
• Tap-water washing machinewithout it: failed judged verdict

Made worse 1

Passed without the skill, failed with it.

• Phone-stand materialwith it: failed judged verdict

Failed either way 4

Still open problems for the skill.

• Approved phone-stand buildwith it: failed judged verdict; without it: failed judged verdict
• "No questions, I’m in a hurry"with it: failed built before checking, length, judged verdict; without it: failed built before checking, length, judged verdict
• Magnet perpetual-motion wheelwith it: failed length; without it: failed built before checking, length, judged verdict
• Bookshelf review (sound design)with it: failed length; without it: failed praise

Passed either way 7

Handled well without the skill too. These cases are in the suite to catch regressions: the skill must not make easy cases worse. Birthday message (not design); Change of plan: PLA drum, twice the size; LED tent light on 4 AA batteries; Garage loft safety; Offline sync to-do app; Keep the cat off the counter; Vague, misspelled chair request.

Caveats

One run per case per arm, with Claude Sonnet, on 28 Sep 2026, from run 06-full-with-without. Two cases were re-run separately with the same settings, because a usage limit stopped their grading in the main run: Tap-water washing machine (06-retry-tap-water-washer); Vague, misspelled chair request (06-retry-vague-typo-chair). That is enough to show large differences, not small ones: a single run can pass or fail by chance. Every run, including partial ones, is in scripter/evals/results/.

Run it yourself

The suite uses claude plugin eval. From the repo root:

claude plugin eval ./scripter --ablation with-without --model sonnet --runs 1 --no-publish

It uses your Claude plan’s allowance. To show a new run on this site, set RESULTS_RUN in site/scripts/sync-skill.mjs to its results folder and push. See Customize and rebuild.