# cdc-lens: CDC Event Analyzer & SCD2 History Builder CLI

## PURPOSE
Before building "history tables," Artie replicated databases one-to-one: whatever the
Postgres table looked like, the destination looked the same. The history-table
feature changed that — it emits an SCD Type 2 stream where updating a row ten times
produces ten rows, each with a timestamp and the type of operation (insert / update /
delete). `cdc-lens` is a Click + Pandas CLI that ingests a CDC change-event file and
either (a) reports replication health, or (b) reconstructs the SCD2 history table that
Artie would produce.

## WHY THIS MATTERS
The history table is the product's hero story: every customer said they did not need
it, and six months after launch *40% turned it on*. It exists for exactly the reasons
the CLI surfaces — training ML models on historical data, debugging, auditing who
changed what and when. This project makes that mechanism concrete and inspectable.

## HOW TO BUILD
1. `cdc-lens stats EVENTS.jsonl` — parse events with Pandas, print per-table event
   counts, operation mix, out-of-order count and end-to-end lag percentiles.
2. `cdc-lens build EVENTS.jsonl --table orders` — fold events into an SCD2 frame:
   `valid_from`, `valid_to`, `is_current`, `op`.
3. Validate: `is_current` rows must equal distinct primary keys; no overlapping
   `valid_from`/`valid_to` windows.
4. `cdc-lens export ... --format parquet` for a downstream warehouse sink.

```python
@click.group()
def cli(): ...

@cli.command("build")
@click.argument("events", type=click.Path(exists=True))
@click.option("--table", required=True)
def build(events, table):
    df = pd.read_json(events, lines=True)
    history = to_scd2(df[df.table == table])
    click.echo(history.to_string(index=False))
```

## WHEN TO USE IN PRODUCTION
Use it to backfill history tables, audit pipeline changes, or reconstruct state for
ML feature training and debugging. It is ideal for local/offline warehousing work and
for validating that a replication engine's SCD2 output is correct. Do NOT use it as a
live streaming processor — it is batch/file-oriented. If you only ever need the
current row, skip SCD2 entirely; it trades storage and write volume for history.

## HOW IT HELPS STUDENTS
You learn Click command design, real Pandas windowing/dedup logic, and the SCD Type 2
data-modeling pattern used across every analytical warehouse. You also practise
turning an ambiguous product idea into a crisp, testable transformation.

## STARTER CODE
`starter/main.py` contains the Click group, a deterministic event parser and a
partially-implemented `to_scd2` with `# TODO` markers for windowing and validation.
