Fernwick Air: building the scored set

workflow.yaml is AWML 0.6, as it was pushed. The registry draws, checks and fits it converted to AWML 0.8.

awml: "0.6"
title: "Fernwick Air: building the scored set"
description: |
  Each week, for one intent, code samples closed cases. Two people score each case blind against the
  rubric's yes or no questions, a third settles any disagreement, and agreed cases go into the scored
  set (the gold set), with how often the two scorers agreed. Fernwick Air is fictional.
scopes:
  - { id: tokenised, label: "tokenised: Fernwick staff", processing: { anthropic: any } }
  - { id: ops, label: "ops: Fernwick staff", processing: none, in: tokenised }
parts:
  - { id: weekly, kind: trigger, label: "Each week, for the intent being scored", wait: P1D }
  - { id: closed, kind: context, label: "Closed cases, tokenised: what was asked, what the clerk answered, what was sent and done", in: tokenised, untrusted: true, ref: chrismdp/counting-house-live#closed }
  - { id: sample, kind: code, op: bring-in, label: "Sample a few hundred closed cases, oversampling the risky and the rare", in: tokenised }
  - { id: split, kind: fork, in: tokenised }
  - { id: scorer_a, kind: person, op: check, label: "The first scorer answers the rubric's questions, blind", in: tokenised }
  - { id: scorer_b, kind: person, op: check, label: "The second scorer answers the rubric's questions, blind", in: tokenised }
  - { id: compare, kind: code, op: check, label: "Once both have scored, compare their answers", in: tokenised }
  - { id: settle, kind: person, op: check, label: "A third person settles each disagreement", in: tokenised }
  - { id: file_gold, kind: code, op: do, label: "File each agreed case in the scored set, and the scorers' agreement", in: tokenised }
  - { id: gold, kind: context, label: "The scored set: real cases, each with its true intent and agreed answers, a slice held out for exams", in: tokenised }
  - { id: agreement, kind: context, label: "How often the scorers agree, per intent", in: tokenised }
lines:
  - { from: weekly, to: sample, kind: start }
  - { from: closed, to: sample, kind: data }
  - { from: sample, to: split, kind: start }
  - { from: split, to: scorer_a, kind: start }
  - { from: split, to: scorer_b, kind: start }
  - { from: closed, to: scorer_a, kind: data, label: the sampled cases }
  - { from: closed, to: scorer_b, kind: data, label: the sampled cases }
  - { from: scorer_a, to: compare, kind: start, label: their answers }
  - { from: scorer_b, to: compare, kind: start, label: their answers }
  - { from: compare, to: file_gold, kind: start, label: "[agreed]" }
  - { from: compare, to: settle, kind: start, label: "[disagreed]" }
  - { from: settle, to: file_gold, kind: start, label: the settled answers }
  - { from: closed, to: file_gold, kind: data, label: the cases }
  - { from: file_gold, to: gold, kind: data }
  - { from: file_gold, to: agreement, kind: data }