Fernwick Air: testing the judge

workflow.yaml is AWML 0.6, as it was pushed. The registry draws, checks and fits it converted to AWML 0.8.

awml: "0.6"
title: "Fernwick Air: testing the judge"
description: |
  Whenever the judge's model, prompt or rubric changes, it scores the held-out slice of the scored
  set. Code compares how often it agrees with the people against how often the people agree with
  each other, and records the intents the judge may be trusted on. Fernwick Air is fictional.
scopes:
  - { id: tokenised, label: "tokenised: Fernwick staff", processing: { anthropic: any } }
  - { id: ops, label: "ops: Fernwick staff", processing: none, in: tokenised }
parts:
  - { id: changed, kind: trigger, label: "The judge's model, prompt or rubric changes", wait: P1D }
  - { id: gold, kind: context, label: "The scored set: real cases, each with its true intent and agreed answers, a slice held out for exams", in: tokenised, untrusted: true, ref: chrismdp/counting-house-gold#gold }
  - { id: agreement, kind: context, label: "How often the scorers agree, per intent", in: tokenised, ref: chrismdp/counting-house-gold#agreement }
  - { id: judge, kind: ai, op: check, label: "The judge scores the held-out cases, without their agreed answers", in: tokenised }
  - { id: agree, kind: code, op: check, label: "Per intent, does the judge agree with the people about as often as they agree with each other?", in: tokenised }
  - { id: trust, kind: code, op: do, label: "Record the intents the judge is trusted on", in: tokenised }
  - { id: judge_trust, kind: context, label: "The intents the judge is trusted on", in: tokenised }
lines:
  - { from: changed, to: judge, kind: start }
  - { from: gold, to: judge, kind: data, label: the held-out cases }
  - { from: judge, to: agree, kind: start, label: its scores }
  - { from: gold, to: agree, kind: data, label: the agreed answers }
  - { from: agreement, to: agree, kind: data }
  - { from: agree, to: trust, kind: start, label: the verdict per intent }
  - { from: trust, to: judge_trust, kind: data }