148 lines
4.1 KiB
Markdown
148 lines
4.1 KiB
Markdown
# Test Results
|
|
|
|
These are synthetic checks for the skill. They are not proof of style match, but they catch the common failures.
|
|
|
|
Use `TEST_SET.md` for held-out examples. This file records representative pass/fail patterns and scoring expectations.
|
|
|
|
## Test 1: Preserve Blunt Technical Critique
|
|
|
|
Input:
|
|
|
|
> this approach is probably wrong, you're hiding the hard bit in the queue and calling it done
|
|
|
|
Bad output:
|
|
|
|
> There may be some architectural concerns with the current approach, particularly around how queueing responsibilities are being handled.
|
|
|
|
Score: 3/10.
|
|
|
|
Why it fails: softened judgement, fake consultancy phrasing, less specific than the draft.
|
|
|
|
Accepted output:
|
|
|
|
> This approach is probably wrong. You're hiding the hard bit in the queue and calling it done.
|
|
|
|
Score: 9/10.
|
|
|
|
Why it passes: minimal edit, same rhythm, same criticism.
|
|
|
|
## Test 2: Agent Instruction
|
|
|
|
Input:
|
|
|
|
> make the tests stop bodging the db. use the actual postgres test db and don't mock the part we're trying to test
|
|
|
|
Bad output:
|
|
|
|
> It would be good to improve the database testing strategy by using a more realistic Postgres-backed approach where appropriate.
|
|
|
|
Score: 2/10.
|
|
|
|
Why it fails: vague, polite, removes irritation, hides the concrete instruction.
|
|
|
|
Accepted output:
|
|
|
|
> Stop bodging the DB tests. Use the actual Postgres test DB and don't mock the part we're trying to test.
|
|
|
|
Score: 9/10.
|
|
|
|
Why it passes: direct imperative, preserves the point and the edge.
|
|
|
|
## Test 3: Formal-Public Roundup
|
|
|
|
Input:
|
|
|
|
> harvester jobs now timeout after 10 minutes. catwalk deploys were leaving jobs stuck as running for an hour
|
|
|
|
Bad output:
|
|
|
|
> Improved Harvester operational resilience by implementing stricter timeout handling for synchronization workloads.
|
|
|
|
Score: 2/10.
|
|
|
|
Why it fails: release-note sludge, loses the actual user/team impact.
|
|
|
|
Accepted output:
|
|
|
|
> Harvester jobs now time out after 10 minutes. Catwalk deploys were leaving jobs stuck as running for an hour.
|
|
|
|
Score: 9/10.
|
|
|
|
Why it passes: keeps the concrete change and why it mattered.
|
|
|
|
## Test 4: Debugging State Update
|
|
|
|
Input:
|
|
|
|
> codex now spawns, but auth is broken.
|
|
|
|
Bad output:
|
|
|
|
> Codex process initialization is now working, but authentication appears to be experiencing issues.
|
|
|
|
Score: 4/10.
|
|
|
|
Why it fails: support-ticket phrasing, weaker than the source, no benefit from the extra words.
|
|
|
|
Accepted output:
|
|
|
|
> Codex now spawns, but auth is broken.
|
|
|
|
Score: 10/10.
|
|
|
|
Why it passes: only fixes casing. Keeps the terse state transition.
|
|
|
|
## Test 5: Casual Opinion
|
|
|
|
Input:
|
|
|
|
> feels like too many files, need a happy medium
|
|
|
|
Bad output:
|
|
|
|
> The current structure may be over-fragmented and could benefit from a more balanced organisation.
|
|
|
|
Score: 3/10.
|
|
|
|
Why it fails: consultant phrasing, loses the rough architecture instinct.
|
|
|
|
Accepted output:
|
|
|
|
> Feels like too many files. Need a happy medium.
|
|
|
|
Score: 9/10.
|
|
|
|
Why it passes: minimal punctuation edit, preserves the phrase that matters.
|
|
|
|
## Test 6: Formal-Public Longform
|
|
|
|
Input:
|
|
|
|
> If TeamSpeak as a company died tomorrow, my server would still be running. My client application might not receive any more updates, but I could keep using it. Same can't be said for Discord.
|
|
|
|
Bad output:
|
|
|
|
> Self-hosted voice infrastructure provides more resilience than centrally managed platforms because it can continue operating independently of the vendor.
|
|
|
|
Score: 2/10.
|
|
|
|
Why it fails: turns a concrete argument into an abstract product principle.
|
|
|
|
Accepted output:
|
|
|
|
> If TeamSpeak died tomorrow, my server would still be running. My client might not get updates anymore, but I could keep using it. Same can't be said for Discord.
|
|
|
|
Score: 8/10.
|
|
|
|
Why it passes: slightly tighter but keeps the scenario and blunt contrast.
|
|
|
|
Observed risks:
|
|
|
|
- Models still try to add "worth noting" and "it may be beneficial" unless `ANTI_STYLE.md` is followed.
|
|
- Models over-structure short instructions into headings and summaries.
|
|
- Models remove mild irritation, which makes the text less plausible.
|
|
- Models turn roundups into release notes unless `formal-public` examples are used.
|
|
- Models turn concrete public arguments into abstract principles unless checked against `TEST_SET.md`.
|
|
|
|
Use `EVALUATOR.md` before final output.
|