# Managed Vector Search Sync Simulator (SimPy + NumPy)

## PURPOSE
Step one of the Agent Bricks demo is a vector search index built over a Delta
table with **managed embeddings** — "managed is the keyword here because you
don't have to set up a separate embedding service and the index itself keeps in
sync with the table as the data changes." This project is a discrete-event
simulation of that sync pipeline: Delta-table writes arrive, embedding work is
queued, and the index is updated. It measures sync lag and staleness at query
time under different write loads.

## WHY THIS MATTERS
The demo asserts "the index keeps in sync with the table" as a fact, but
in-sync is a property you have to *earn* under load. A simulation teaches the
trade-offs: a write-heavy product catalog can push embedding jobs into a
backlog, so a query issued right after an update may read a stale index. This
is exactly the retrieval step the MLflow trace later shows — if retrieval is
slow or stale, the trace is where you'd notice.

## HOW TO BUILD
1. Model the system with SimPy: a process for table writes, a bounded pool of
   embedding workers, and query arrivals.
2. Track per-row `written_at` and `indexed_at`; staleness is the difference at
   query time.
3. Use NumPy for the arrival process (Poisson writes via a seeded RNG) and for
   summarising lag percentiles.
4. Sweep worker counts to find where the queue stops growing.

Key snippet — the embedding worker consuming a write queue:

```python
def indexer(env, queue, workers, stats, embed_latency):
    def worker(_):
        while True:
            row = yield queue.get()
            yield env.timeout(embed_latency)
            row["indexed_at"] = env.now
            stats.append(env.now - row["written_at"])
    for i in range(workers):
        env.process(worker(i))
```

## WHEN TO USE IN PRODUCTION
Use it when sizing or capacity-planning a sync pipeline: how many embedding
workers keep index lag under a target during a catalog import, or whether a
bursty write pattern causes unacceptable staleness. Do NOT use it as a
correctness oracle for the real product — real managed sync also handles
deletes, schema changes and retries. It is a planning model, not a test double.

## HOW IT HELPS STUDENTS
Students learn event-driven simulation with SimPy, seeded stochastic arrival
modelling with NumPy, and how to reason about queueing and staleness in a
retrieval system. They connect a scheduling abstraction (workers, queues) to a
product requirement ("the index stays in sync") and practise reading percentile
lag rather than a single average.

## STARTER CODE
See `starter/main.py` for a runnable SimPy model with a CLI entry point, a
seeded RNG, printed lag percentiles, and `# TODO` markers for worker sweeps.
