Fernwick Air: demoting a model when its scores fall

awml: "0.8"
title: "Fernwick Air: demoting a model when its scores fall"
description: |
  A fix for a gap in the Counting House: nothing moved a model back down. Every night code compares
  each intent's recent scores with the bar in its policy row. Below the bar, the intent drops one
  tier straight away, and QA and data science find the cause before putting it up for promotion
  again. Fernwick Air is fictional.
scopes:
  - { id: tokenised, label: "tokenised: Fernwick staff", processing: { anthropic: any } }
parts:
  - { id: nightly, kind: trigger, label: "Every night, once the day's scores are recorded", wait: PT6H }
  - { id: scores, kind: context, label: "Every contact's scores, by intent and contact reason", in: tokenised, ref: chrismdp/counting-house-ongoing#scores }
  - { id: policy_table, kind: context, label: "The policy table: each intent's model, tier, SOP, tools and approvers", in: tokenised, ref: chrismdp/counting-house-promotion#policy_table }
  - { id: compare, kind: code, op: check, label: "Compare each intent's last seven days of scores with the bar in its policy row", in: tokenised }
  - { id: drop, kind: code, op: do, label: "Drop the intent one tier, and record the scores that tripped it", in: tokenised }
  - { id: tell, kind: code, op: do, label: "Post the demotion for QA and data science", in: tokenised }
  - { id: alert, kind: surface, label: "Demotions: the intent, its old and new tier, and the scores behind it", in: tokenised }
  - { id: investigate, kind: person, op: do, label: "QA and data science find the cause, fix it, and put the intent up for promotion again", in: tokenised }
  - { id: repromote, kind: outside, label: "Put the intent up for promotion: Ask", in: tokenised, ref: chrismdp/counting-house-promotion#put_up }
lines:
  - { from: nightly, to: compare, kind: start }
  - { from: scores, to: compare, kind: data }
  - { from: policy_table, to: compare, kind: data, label: the bar }
  - { from: compare, to: drop, kind: start, label: "[below the bar]" }
  - { from: drop, to: policy_table, kind: data, label: one tier down }
  - { from: drop, to: tell, kind: start }
  - { from: tell, to: alert, kind: data }
  - { from: tell, to: investigate, kind: start }
  - { from: alert, to: investigate, kind: data }
  - { from: investigate, to: repromote, kind: data, label: "[fixed]" }