awml
Sign in

chrismdp / counting-house-gold

Fernwick Air: building the scored set

Public ·
Version 2 of 3
  1. #3 · chrismdp · · 0.6Latest
  2. #2 · chrismdp · · 0.6Showing
  3. #1 · chrismdp · · 0.6
Fork
Who may see each partWhere each part runsYAMLChecksNo mistakes
tokenisedFernwick staffScopeops: Fernwick staffthe sampled casesthe sampled casestheir answerstheir answers[agreed][disagreed]the settled answersthe casesTriggerEach week, for the intentbeing scoredLinked to chrismdp/counting-house-live#closedData · untrustedClosed cases, tokenised: whatwas asked, what the clerkanswered, what was sent anddoneCode · bring-inSample a few hundred closedcases, oversampling the riskyand the rarePerson · checkThe first scorer answers therubric's questions, blindPerson · checkThe second scorer answersthe rubric's questions, blindCode · checkOnce both have scored,compare their answersPerson · checkA third person settles eachdisagreementCode · doFile each agreed case in thescored set, and the scorers'agreementUsed by chrismdp/counting-house-judge-test and chrismdp/counting-house-promotionDataThe scored set: real cases,each with its true intent andagreed answers, a slice heldout for examsUsed by chrismdp/counting-house-judge-testDataHow often the scorers agree,per intent