# Drift Watch — Weekly Agent Accuracy CLI

## PURPOSE
Siroui Mushegian explains Barracuda's post-production discipline: "we don't let that be the final stop. That's actually the beginning... every month that goes by, every week that goes by, we're measuring them to make sure that they are as accurate as they were the first time that we put them in place." This project is a Click + Pandas command-line tool that ingests weekly agent-accuracy exports and reports exactly that: which deployed AI agents have drifted away from their original baseline, and by how much.

## WHY THIS MATTERS
Most teams measure model quality once, at launch, then move on — the failure mode this transcript explicitly warns against. Drift Watch operationalises "production is the beginning" as a repeatable weekly ritual. It contrasts each agent's current accuracy against the accuracy it posted when first deployed (the "first time we put them in place") and surfaces a ranked list of agents that need attention.

## HOW TO BUILD
Build a Click group with three subcommands operating on a CSV of `agent_id, week, accuracy, volume`:
- `baseline` — capture the first observed accuracy per agent into a baseline JSON.
- `check` — compare latest week vs baseline, compute delta, and print a Rich-style table sorted worst-first.
- `trend` — print a per-agent accuracy series so a student can eyeball the slope.

Pandas does the grouping; Click handles arguments; JSON persists the baseline.

```python
@cli.command()
@click.option("--metrics", type=click.Path(exists=True), required=True)
@click.option("--baseline", type=click.Path(exists=True), required=True)
@click.option("--threshold", default=0.05, show_default=True)
def check(metrics, baseline, threshold):
    df = pd.read_csv(metrics).sort_values("week")
    base = json.loads(Path(baseline).read_text())
    latest = df.groupby("agent_id").last()["accuracy"]
    report = [
        {"agent": a, "baseline": base[a], "latest": float(latest[a]),
         "delta": float(latest[a] - base[a])}
        for a in base if a in latest.index
    ]
    for row in sorted(report, key=lambda r: r["delta"]):
        flag = "DRIFT" if row["delta"] < -threshold else "ok"
        click.echo(f"{row['agent']:<12} {row['baseline']:.3f} -> {row['latest']:.3f}  {flag}")
```

## WHEN TO USE IN PRODUCTION
Use it in a scheduled CI job or cron that runs after each weekly metrics export, alerting an owner when an agent's accuracy falls past a threshold. Ideal for teams running many small agents where manual review does not scale. Do **not** use it as a substitute for labelled evaluation sets, for real-time anomaly detection (it is batch/weekly by design), or when a single global accuracy number hides per-slice failures — extend with segmentation first.

## HOW IT HELPS STUDENTS
Students practise Click command design, Pandas groupby/diff logic, and the discipline of persisting a baseline for comparison — a genuinely under-taught production skill. They finish with a tool they could wire into a real alerting pipeline.

## STARTER CODE
`starter/main.py` implements the Click CLI, Pandas drift math, and baseline round-tripping, with `# TODO` markers for thresholds, Slack alerts, and per-segment checks.
