chrismdp / counting-house-judge-testFernwick Air: testing the judgePublic · Version 1 of 1ForkWho may see each partWhere each part runsYAMLChecksNo mistakestokenisedFernwick staffScopeops: Fernwick staffthe held-out casesits scoresthe agreed answersthe verdict per intentTriggerThe judge's model, prompt orrubric changesLinked to chrismdp/counting-house-gold#goldData · untrustedThe scored set: real cases,each with its true intent andagreed answers, a slice heldout for examsLinked to chrismdp/counting-house-gold#agreementDataHow often the scorers agree,per intentAI · checkThe judge scores the held-outcases, without their agreedanswersCode · checkPer intent, does the judgeagree with the people aboutas often as they agree witheach other?Code · doRecord the intents the judge istrusted onUsed by chrismdp/counting-house-ongoing and chrismdp/counting-house-promotionDataThe intents the judge istrusted onFlow only+−Fit