# MLflow Trace Cost & Latency Analyzer (Click + Pandas)

## PURPOSE
In the Agent Bricks demo, MLflow tracing "captures exactly what happened on
every request": the retrieval step, the tool calls, the latency, the token
count, and the cost per request, laid out span by span. This project is a CLI
that ingests exported trace records and turns them into a per-request table and
aggregate report, so a student can "see what your agent did span by span
instead of squinting at logs and guessing."

## WHY THIS MATTERS
The video's first "moment in the loop" is reading a single trace (it took about
5 seconds) and the second is knowing the whole set is healthy via evaluation.
Traces are the debugging artefact: "when something breaks in production... this
is the first place that you want to look." A CLI that computes latency, tokens,
and cost per request makes that first look fast and repeatable, and gives you
the raw material to compare against MLflow evaluation scores.

## HOW TO BUILD
1. Take a `--traces` JSONL file (one MLflow-style trace per line) and a
   `--cost-per-1k-tokens` rate.
2. Flatten each trace's spans into rows with `span_name`, `duration_ms`, and
   `tokens`, using Pandas.
3. Compute per-request totals (retrieval ms + LLM ms), token counts, and
   estimated cost.
4. Group by span name to see where time goes; flag requests over a latency
   budget.

Key snippet — the aggregation that powers the report:

```python
df = pd.json_normalize(records, "spans", ["request_id"])
df["cost_usd"] = df["tokens"] / 1000 * cost_per_1k
summary = df.groupby("span_name").agg(
    calls=("tokens", "size"),
    avg_ms=("duration_ms", "mean"),
    tokens=("tokens", "sum"),
)
```

## WHEN TO USE IN PRODUCTION
Use it in CI or a nightly job to watch p95 latency and cost creep after every
agent change, or during an incident to triage a batch of exported traces. Do
NOT use it for live, per-request monitoring — that is MLflow's tracing UI and
production monitoring's job. It reads static exports; it is not a streaming
collector.

## HOW IT HELPS STUDENTS
Students practise Click subcommands, Pandas normalisation of nested JSON, and
group-by aggregation on realistic observability data. They internalise the
trace vocabulary from the video (retrieval, tool call, latency, tokens, cost)
and learn to turn raw spans into the cost/latency numbers that decide whether
an agent is production-ready.

## STARTER CODE
See `starter/main.py` for a Click CLI with `summary` and `slow` commands, a
deterministic sample trace loader, and `# TODO` markers for Parquet/CSV input.
