# streamsim: Sub-Minute Replication Lag Simulator

## PURPOSE
Jacqueline Cheong says real-time streaming systems are "unforgiving — very powerful,
but really really hard to get right," and that at OpenDoor it took eight engineers six
quarters to productionize one. `streamsim` is a SimPy + NumPy discrete-event simulator
of a Postgres → change-data-capture → message-queue → warehouse (Snowflake/Databricks/
Iceberg) pipeline. It models producers, CDC extraction, a Kafka-style queue, a
destination sink and a DBLog-style backfill, then reports whether the pipeline holds
Artie's advertised sub-minute freshness.

## WHY THIS MATTERS
The video's wedding-cake metaphor says the hard part was never "moving data A to B" —
it is guaranteeing no dropped rows, no out-of-order events, reliable extraction with
low load on the source, and fast backfill. A simulator lets you stress those
constraints cheaply: crank throughput, inject a burst, kill a consumer, and watch p99
lag and ordering violations change before you ever touch production.

## HOW TO BUILD
1. Producers generate change events at a rate (Poisson via NumPy) per table.
2. A CDC stage lifts events with a per-event extraction cost, capping concurrency.
3. A bounded queue with N consumers replays events to the sink.
4. A backfill process competes for throughput after a simulated outage.
5. Record per-event e2e latency; assert monotonic ordering per primary key.

```python
import simpy

def consumer(env, name, queue, sink, stats):
    while True:
        event = yield queue.get()
        yield env.timeout(sink.replay_cost)      # write to warehouse
        stats.record(event, env.now)             # e2e lag + ordering checks
        queue.consumers.release()

env = simpy.Environment()
env.run(until=600)                                # simulate 10 minutes
print(stats.summary())                            # p50/p95/p99 lag, violations
```

## WHEN TO USE IN PRODUCTION
Use it to size consumer counts, pick queue depth, tune backfill throttling and set
realistic freshness SLOs before committing engineers to a build-out. It is the right
tool for capacity planning and for teaching why "just move data fast" is hard. Do NOT
use it as the real pipeline or as a benchmark of a specific database's performance —
it is an abstract queuing model, and its numbers are only as good as its assumptions.

## HOW IT HELPS STUDENTS
You get hands-on SimPy process modeling, NumPy statistical workload generation, and
queueing/lag analysis — the exact reasoning behind Netflix's DBLog paper and the
internal systems at DoorDash and Instacart. You learn to argue about pipelines with
numbers instead of vibes.

## STARTER CODE
`starter/main.py` builds the SimPy environment, a production stub, consumers and a
stats collector with `# TODO` markers for backfill, burst injection and p99 reporting.
