4.1 KiB
Test Results
These are synthetic checks for the skill. They are not proof of style match, but they catch the common failures.
Use TEST_SET.md for held-out examples. This file records representative pass/fail patterns and scoring expectations.
Test 1: Preserve Blunt Technical Critique
Input:
this approach is probably wrong, you're hiding the hard bit in the queue and calling it done
Bad output:
There may be some architectural concerns with the current approach, particularly around how queueing responsibilities are being handled.
Score: 3/10.
Why it fails: softened judgement, fake consultancy phrasing, less specific than the draft.
Accepted output:
This approach is probably wrong. You're hiding the hard bit in the queue and calling it done.
Score: 9/10.
Why it passes: minimal edit, same rhythm, same criticism.
Test 2: Agent Instruction
Input:
make the tests stop bodging the db. use the actual postgres test db and don't mock the part we're trying to test
Bad output:
It would be good to improve the database testing strategy by using a more realistic Postgres-backed approach where appropriate.
Score: 2/10.
Why it fails: vague, polite, removes irritation, hides the concrete instruction.
Accepted output:
Stop bodging the DB tests. Use the actual Postgres test DB and don't mock the part we're trying to test.
Score: 9/10.
Why it passes: direct imperative, preserves the point and the edge.
Test 3: Formal-Public Roundup
Input:
harvester jobs now timeout after 10 minutes. catwalk deploys were leaving jobs stuck as running for an hour
Bad output:
Improved Harvester operational resilience by implementing stricter timeout handling for synchronization workloads.
Score: 2/10.
Why it fails: release-note sludge, loses the actual user/team impact.
Accepted output:
Harvester jobs now time out after 10 minutes. Catwalk deploys were leaving jobs stuck as running for an hour.
Score: 9/10.
Why it passes: keeps the concrete change and why it mattered.
Test 4: Debugging State Update
Input:
codex now spawns, but auth is broken.
Bad output:
Codex process initialization is now working, but authentication appears to be experiencing issues.
Score: 4/10.
Why it fails: support-ticket phrasing, weaker than the source, no benefit from the extra words.
Accepted output:
Codex now spawns, but auth is broken.
Score: 10/10.
Why it passes: only fixes casing. Keeps the terse state transition.
Test 5: Casual Opinion
Input:
feels like too many files, need a happy medium
Bad output:
The current structure may be over-fragmented and could benefit from a more balanced organisation.
Score: 3/10.
Why it fails: consultant phrasing, loses the rough architecture instinct.
Accepted output:
Feels like too many files. Need a happy medium.
Score: 9/10.
Why it passes: minimal punctuation edit, preserves the phrase that matters.
Test 6: Formal-Public Longform
Input:
If TeamSpeak as a company died tomorrow, my server would still be running. My client application might not receive any more updates, but I could keep using it. Same can't be said for Discord.
Bad output:
Self-hosted voice infrastructure provides more resilience than centrally managed platforms because it can continue operating independently of the vendor.
Score: 2/10.
Why it fails: turns a concrete argument into an abstract product principle.
Accepted output:
If TeamSpeak died tomorrow, my server would still be running. My client might not get updates anymore, but I could keep using it. Same can't be said for Discord.
Score: 8/10.
Why it passes: slightly tighter but keeps the scenario and blunt contrast.
Observed risks:
- Models still try to add "worth noting" and "it may be beneficial" unless
ANTI_STYLE.mdis followed. - Models over-structure short instructions into headings and summaries.
- Models remove mild irritation, which makes the text less plausible.
- Models turn roundups into release notes unless
formal-publicexamples are used. - Models turn concrete public arguments into abstract principles unless checked against
TEST_SET.md.
Use EVALUATOR.md before final output.