Writing a dataset

File layout

The folder name, the name in SKILL.md, and skill in evals/cases.yaml are the same string.

json-response/
  SKILL.md
  scripts/agent_entry.py
  evals/
    cases.yaml
    fixtures/
      orders.json
      empty-orders.json
    goldens/
      flatten-orders.json
      edge-empty.json
    history/

Copy this pack

Copy these four files into skills/json-response/. This is the empty-input case from that example. The pack on disk adds two more cases. Then run skillgate ingest skills/json-response.

SKILL.md

---
name: json-response
description: Reshape a JSON API response. Flatten nested fields to id, status, and total, and drop empty values. Use when the user has a JSON response.
---

# JSON response

Flatten each object under `data` to `id`, `status`, and `total`. An empty `data` array becomes `[]`.

evals/fixtures/empty-orders.json

{
  "data": []
}

evals/goldens/edge-empty.json

[]

evals/cases.yaml

schema: 1
skill: json-response
cases:
  - id: edge-empty
    status: pinned
    prompt: Flatten the JSON API response in empty-orders.json. Save output/orders.json.
    should_trigger: true
    input:
      files:
        - src: evals/fixtures/empty-orders.json
          dest: empty-orders.json
    golden:
      output_path: output/orders.json
      path: evals/goldens/edge-empty.json
      kind: json
    contains:
      - "[]"

status: pinned is what runs. dest is the name the agent sees.golden.output_path is the file the agent must write. golden.path is the expected file in the pack. kind is text, json, or bytes. contains lists substrings that must appear. should_trigger: false is an unrelated prompt: leave input.files empty and set golden: null.

What each example tests

• json-response: flatten two orders; an empty list becomes []; a pytest prompt should not trigger.
• readme-writer: a README from brief.txt; a missing license renders not specified.
• log-summarizer: two timeouts and one warning; a quiet log is Errors: 0.
• sql-writer: id and total for open orders; asking for customer_email when the schema does not have that column.
• receipt-extractor: merchant, date, and total 18.40 from a fictional receipt; a receipt with no total stays null.

json-response-sabotaged copies the JSON cases. Its instructions ignore JSON and write a snack list, so a live check fails the trigger cases. The unrelated case can still pass.

How to write a case

Start with 3 to 6 pinned cases. Each example ships three: one main behavior, one edge, and one unrelated prompt.

• One behavior per case. flatten-orders checks the flat list. edge-empty checks [].
• Realistic inputs. Keep the file small enough to finish from. harbor-receipt.txt is a fictional shop receipt.
• Clear expected outcomes. Put the full file in evals/goldens/ and the required lines in contains.
• An edge case. Empty, missing, or zero. log-summarizer uses a log with no errors.
• One case that fails when the skill is wrong. Set should_trigger: false on a prompt from a different job. To see a whole pack fail, check json-response-sabotaged. skillgate check skills/json-response-sabotaged --expect-fail exits 0 only when that run is red.
• No answer leakage. The receipt prompt says to extract the total. It does not contain 18.40. The README edge prompt says not to invent a license. It does not contain not specified.
• Small files, no personal data. Each example file is under 4 KB. Use fictional names. The receipt merchant is Harbor Paper Co.

Check it locally

skillgate ingest skills/json-response
skillgate case list skills/json-response
skillgate check skills/json-response
skillgate check skills/json-response --case flatten-orders
skillgate check skills/json-response-sabotaged
skillgate check skills/json-response-sabotaged --expect-fail

skillgate ingest prints OK json-response when SKILL.md loads. skillgate case list prints each id, pinned, and the prompt. skillgate check exits 0 when the live run is green, 1 when it is red, and 2 when no agent is available, you are not signed in, or no decision model is selected. Sign in with skillgate login, then pick a decision model. Without those, the check exits 2. That is not a pass.

skillgate ui

1. Click Skills. The page title is Skill Repository.
2. Click Examples, then Install.
3. Open json-response. Click Evaluate this skill.
4. Click Run. The label is Run 3 cases when three checks are loaded. Until you sign in or add a key, the line above the button is Next: sign in or add a key in Settings, then press Run.
5. For the red path, open json-response-sabotaged and press Run. The JSON checks fail. The unrelated check can pass.

Common mistakes

• The name does not match the folder. Use one slug for the folder, name in SKILL.md, and skill in cases.yaml.
• skillgate case list shows the case and skillgate check skips it. Set status: pinned.
• The fixture is missing. src is relative to the skill folder, for example evals/fixtures/orders.json.
• A .md brief never shows up as an input. Case inputs use a data suffix. readme-writer stores the brief as brief.txt. Expected READMEs stay under evals/goldens/.
• The golden is not found. golden.path points at evals/goldens/<file>. output_path is where the agent writes, usually output/<file>.
• The prompt already contains the answer. Move that sentence into the golden and contains. In receipt-extractor case extract-total, 18.40 is in the fixture and the golden, not in the prompt.
• Every case is the happy path. Add the edge and the should_trigger: false case.
• The pack is huge, or it contains a real email or phone number. Shrink the fixture and use a fictional value, the way harbor-receipt.txt does.
• skillgate examples install prints Left N existing file(s) in place. and your edit is still there. That command never replaces a file that already exists. Delete the file you want replaced, or install into an empty folder with --dest.