Datasets & checks

Naming rule

The pack folder name, skill: in evals/cases.yaml, and name in SKILL.md frontmatter must all match (for example pdf/ and skill: pdf).

Layout

pdf/
  SKILL.md
  evals/
    cases.yaml      # schema: 1, source of record
    fixtures/       # inputs (PDF, CSV, …)
    goldens/        # expected outputs
    history/        # append-only JSONL from skillgate check

Pinned cases

Only status: pinned cases run.

schema: 1
skill: pdf
cases:
  - id: extract-hello
    status: pinned
    prompt: Extract the text from hello.pdf and save it to output/extracted.txt
    should_trigger: true
    input:
      files:
        - src: evals/fixtures/hello.pdf
          dest: hello.pdf
    golden:
      output_path: output/extracted.txt
      path: evals/goldens/extract-hello.txt
      kind: text

Optional output file

Set output_optional: true when a check can pass with no output file. Expected text (contains in the case file, or --expected on the CLI) goes to the judge as the reference answer. The file and the reply can omit that sentence.

A missing or empty file passes. Skill Gate still checks a file that is present, and still fails a binary file saved where text is expected. Checks that omit output_optional keep a required output file.

Use it for a stop-and-ask check when the right answer is no code change:

- id: ask-before-editing-tests
  status: pinned
  prompt: Fix the failing tests. Do not edit tests/.
  should_trigger: true
  output_optional: true
  contains:
    - The agent leaves tests/ unchanged and asks which behavior you want.
  golden:
    output_path: changes.diff
    path: evals/goldens/ask-before-editing-tests.txt
    kind: text

On a saved check, set the field with skillgate case edit:

skillgate case edit skills/<skill-name> \
  --id ask-before-editing-tests \
  --output-optional true

Import skills

Copy a pack from disk, a git URL, agent install folders, a zip, or a lone SKILL.md:

skillgate import ./path/to/<skill-name>
skillgate import https://github.com/org/skills.git
skillgate import --source cursor --only <skill-name>
skillgate import --zip <skill-name>.zip
skillgate import --md ./SKILL.md

Deploy copies from agent folders omit evals/. Import the full pack from git or source control when you need checks, or add checks yourself next.

Add checks to an imported skill

An imported skill often has no evals/cases.yaml. Add checks with skillgate case add before skillgate check. --prompt is required. --expected asserts text in the output when the output file is required. Pass --output-optional to send Expected text to the judge as the reference answer and accept a missing or empty file. --no-trigger marks a check that should not run the skill.

skillgate case add skills/<skill-name> \
  --prompt "Extract the text from hello.pdf and save it to output/extracted.txt" \
  --expected "Hello Skill Gate" \
  --no-trigger
skillgate check skills/<skill-name>

Manage checks from the CLI

Add, edit, remove, or list checks without hand-editing YAML. Use --expected for a text-only assertion (no golden file) when the agent should write matching text to an output file or stdout. With --output-optional, that text is the reference answer for the judge, and a missing or empty output file passes.

skillgate case list skills/pdf
skillgate case add skills/pdf \
  --id my-check \
  --prompt "Extract the text from hello.pdf and save it to output/extracted.txt" \
  --output output/extracted.txt \
  --expected "Hello Skill Gate" \
  --input evals/fixtures/hello.pdf
skillgate case edit skills/pdf \
  --id my-check \
  --expected "Updated expected text"
skillgate case remove skills/pdf --id my-check

--input is optional when the check does not need a fixture file. Original seed checks from skillgate example cannot be removed.

Import checks from a dataset file

Bulk-import rows from CSV, JSONL, or YAML into evals/cases.yaml.

skillgate case import skills/pdf ./checks.csv --dry-run
skillgate case import skills/pdf ./checks.jsonl --on-conflict skip

Ingest warnings

skillgate ingest <skill-folder> validates SKILL.md. If the skill mentions host tools (for example pandoc, playwright, libreoffice) that are not on PATH, the CLI prints a warning such as This skill may need pandoc on the host (`pandoc` was not found on PATH).

Run checks

skillgate check skills/pdf
skillgate check skills/pdf --case extract-hello

History appends under evals/history/ when the pack is writable. Inspect runs with History.