Judge

A decision model scores each live check, and one is required. Skill Gate does not ship a model. Sign in with skillgate login (or store an agent key), then pick one option below. Save that key with skillgate keys set. The key names are on Keys.

OpenRouter

Save an OpenRouter key as the Jev key, JEV_API_KEY.

Fields: base URL https://openrouter.ai/api/v1, model typesafe/jev-1.14, and the API key.

shell

printf '%s\n' "$JEV_API_KEY" | skillgate keys set JEV_API_KEY
skillgate judge provider set jev --base-url https://openrouter.ai/api/v1 --model typesafe/jev-1.14

skillgate judge provider set jev fills that URL and model when you leave the flags off.

In the local UI, run skillgate ui, open Settings, choose OpenRouter, and confirm Advanced shows that URL and model. Paste the key, Save, then Test.

OpenAI

Save an OpenAI key as OPENAI_API_KEY. This option needs the API key only.

shell

printf '%s\n' "$OPENAI_API_KEY" | skillgate keys set OPENAI_API_KEY
skillgate judge provider set openai

In the local UI, run skillgate ui, open Settings, choose OpenAI, paste the key, Save, then Test.

Cloudflare

Save a Cloudflare token as CLOUDFLARE_API_TOKEN. The account id is the id in the account URL. Replace accountid with that id. Letters and digits only.

Fields: account id, model (clef or clef-flash), and the token.

shell

printf '%s\n' "$CLOUDFLARE_API_TOKEN" | skillgate keys set CLOUDFLARE_API_TOKEN
skillgate judge provider set clef --account-id accountid --model clef

In the local UI, run skillgate ui, open Settings, and choose Cloudflare. Enter the account id, pick clef or clef-flash, paste the token, Save, then Test.

Custom endpoint

Save a key for the URL you enter as CUSTOM_DECISION_API_KEY. Fields: base URL, model, and the API key. There is no fixed model list. Replace MODEL with the model name.

shell

printf '%s\n' "$CUSTOM_DECISION_API_KEY" | skillgate keys set CUSTOM_DECISION_API_KEY
skillgate judge provider set custom --base-url https://example.com/v1 --model MODEL

In the local UI, run skillgate ui, open Settings, and choose Custom endpoint. Enter the base URL and model, paste the key, Save, then Test.

Test the connection

shell

skillgate judge test

skillgate judge provider test runs the same check. Neither command prints the key.

See the selection

shell

skillgate judge provider show

skillgate judge provider show prints the option id, the model, and the base URL. skillgate judge provider set prints those same lines after it saves.

shell

the judge
jev
typesafe/jev-1.14
https://openrouter.ai/api/v1

When judge test fails

Match the message, then fix that option:

This API key was rejected. The key is wrong for this option, or it was revoked. Create a new key from the same page and run skillgate keys set for that option again.
This key is for a different endpoint. The URL is not a scoring URL. OpenRouter uses https://openrouter.ai/api/v1.
This account is out of credits. Add credits on that account, then run skillgate judge test.
This looks like an OpenRouter key. Switch to OpenRouter with skillgate judge provider set jev. Run that command, or press Switch.
OpenAI does not use a base URL. It only needs an API key. Run skillgate judge provider set openai.
Cloudflare scores only with clef or clef-flash. Choose OpenRouter for typesafe/jev-1.14. That option does not score typesafe/jev-1.14. Use --model clef or --model clef-flash, or switch to OpenRouter.
OpenRouter scores only with typesafe/jev-1.14. Choose Cloudflare for clef or clef-flash. Use --model typesafe/jev-1.14, or switch to Cloudflare and pick clef or clef-flash.
This decision model needs an API key. Add it in Settings, or with skillgate keys set. Save the key named above.
This decision model needs a base URL. Pass --base-url. OpenRouter's is https://openrouter.ai/api/v1.
This decision model needs an account id. Set it with skillgate judge provider set and --account-id. Cloudflare needs that flag.
An account id uses letters and digits only. A hyphen is not a missing account id.
That account id wasn't found. That account id does not exist. The message is not This key is for a different endpoint.
gpt-4o isn't a decision model. A chat model on a custom endpoint is refused.
This decision model needs a model. Pass --model, or pick one from the list.
Pick a decision model before running a live check. Run one skillgate judge provider set command above.

Scoring errors do not include HTTP codes. A 404 or 405 from a non-scoring URL says This key is for a different endpoint. A bad endpoint, a rejected key, or an empty balance fails that check. Chat and Responses completion models are not judges. Setting the judge backend to responses, chat, or completions fails the run.

unset, auto, or skip selects no decision model. A live check exits 2 and asks you to pick one. A saved key does not turn scoring on. auto clears the selection.

A low-confidence pass still exits 0. You will see: The judge wasn't very sure about this one. That warning does not change exit 0 or 1.

Optional output as a reference answer

When a check sets output_optional: true, Expected text is the reference answer for the judge. A missing or empty output file passes. Skill Gate still checks a file that is present. The case field and a stop-and-ask example are on Datasets & checks.

Run without a decision model

Pass --no-judge only when you mean to run without a decision model. File checks can still exit 0. Exit 1 when the run is red. Exit 2 when no agent is available, you are not signed in, or no decision model is selected, unless you pass --no-judge.

shell

skillgate check skills/json-response --no-judge

Extra questions per skill

Add a scoring question on top of the built-in questions. Built-in questions stay. They can be Warn only.

shell

skillgate judge add-question skills/pdf \
  --ask "Does the output follow the brief?" \
  --type score

--type accepts score (numeric rubric), pick one (pass --choices one or more times), or Yes/No. List configured questions with skillgate judge show skills/pdf.