awml
Sign in

chrismdp / counting-house-promotion

Fernwick Air: promoting a model on one intent

Public ·
Version 1 of 2
  1. #2 · chrismdp · · 0.8Latest
  2. #1 · chrismdp · · 0.6Showing
Fork
Who may see each partWhere each part runsYAMLChecksNo mistakes
tokenisedFernwick staffScopeops: Fernwick staffthe held-out casesthe intent's SOPthe agreed answersits gradesthe people's own mistakesthe intent's live and shadowscores[cheaper than the people][yes][yes][yes][yes][yes]Started by chrismdp/counting-house-demotionTriggerSomeone puts a candidatemodel up for an intent and tierLinked to chrismdp/counting-house-gold#goldData · untrustedThe scored set: real cases,each with its true intent andagreed answers, a slice heldout for examsUsed by chrismdp/counting-house-demotion and chrismdp/counting-house-liveDataThe policy table: each intent'smodel, tier, SOP, tools andapproversAI · draftThe candidate answers eachheld-out case, as the clerkwouldDataThe candidate's examanswersLinked to chrismdp/counting-house-judge-test#judge_trustDataThe intents the judge istrusted onAI · checkThe judge grades thecandidate's answers againstthe agreed onesUsed by chrismdp/counting-house-price-keeperDataFinance's cost for eachconfusion-matrix cellLinked to chrismdp/counting-house-ongoing#scoresDataEvery contact's scores, byintent and contact reasonCode · checkOn an intent the judge istrusted on, count thecandidate's mistakes byconfusion cell, price them, andcompare with the people'sCode · doAsk to change the intent'spolicy rowPerson · reviewFinance's yes to the exactchange: costs and approvalthresholdsPerson · reviewLegal and compliance's yes:may this intent reach this tierPerson · reviewCS ops' yes: the SOP and thefloor are readyPerson · reviewQA and data science's yes: theeval and the judgePerson · reviewPlatform's yes: the tools,gateway and orchestratorOutside actionChange the intent's policy rowAsk