# Memory Case and Replay Lab

MemPlumb v0.13.2 turns reviewed production evidence into durable **Memory Cases**, compares baseline and candidate policies in persisted **Replay Runs**, freezes complete production Write Quality populations into durable **Replay Cohort Plans**, and rechecks that a frozen population is still current before it can control deployment. The objective is operational: a production memory failure becomes a reusable regression asset that can block an unsafe release instead of disappearing inside a trace.

The default scope is `memory_state_retrieval.v1`. It replays candidate-pool generation, the versioned logical candidate-index contract, scoring, eligibility filters, semantic similarity, and Context budget selection over a contemporaneous Memory State Snapshot. It does not replay the historical physical ANN query plan, extraction, or write policy. The legacy `frozen_retrieval.v1` scope remains available for older Context Runs that have no state capture.

| Surface                          | Durable object                                                      | Restricted content                                                     | Required scope                                                       |
| -------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Replay Lab configuration         | Active policy, defaults, supported scopes, coverage                 | No Case payload or state                                               | `evaluation:read`, `evaluation:execute`, or `evaluation:export`      |
| Memory Case list/create response | Opaque Case ID, type, rubric, counts, coverage                      | No content/state fingerprint, query, Memory value, or source ID        | Read scopes above; create requires `evaluation:execute`              |
| Memory Case detail               | Content/state fingerprints, portable payload, source provenance     | Query, Memory values, Actor/Context/adjudication/evaluator references  | `evaluation:export`                                                  |
| Replay Run/list/results          | Policies, thresholds, aggregate gate, alias-based pool/score delta  | No underlying query, Memory values, or raw global IDs                  | Read scopes above; create/cancel/resume require `evaluation:execute` |
| Replay Cohort Plan/list/detail   | Frozen population counts, progress, shards, Run IDs, hashes         | No member list, eligibility snapshot, Case source, or evaluator detail | Read scopes above; create/cancel/resume require `evaluation:execute` |
| Replay Cohort freshness          | Current/changed counts, age gate, validity window, integrity hashes | No added, removed, changed, or unchanged Case IDs                      | Read scopes above                                                    |

## From a Context to a Memory State Snapshot

`MemoryPipeline.context` asks the Store for the candidate result and Replay State together. SQLite captures both under one writer transaction; PostgreSQL uses one repeatable-read transaction. This contemporaneous boundary matters: reconstructing an Actor's current Memories later cannot prove which revisions and index state were visible when the original candidate pool was generated.

The capture produces two restricted objects:

- `memory-state-snapshot-v1` is a content-addressed, revision-exact Actor state with stable identity ordinals. It is capped at 10,000 Memories and 32 MiB. Raw vectors are forbidden.
- `context-replay-state-v1` binds that state to system/effective time, the observed candidate order and source channels, the observed Context Policy, a versioned logical executor identity, and bounded physical-index provenance.

The Store persists the Context Run, complete candidate evidence, state snapshot, and binding atomically. Content-addressed snapshots may be reused by multiple Context Runs for the same Actor. A Workspace can retain at most 256 MiB of captured state; capacity failure is explicit in Context retrieval metadata and does not make the online Context request fail. A Context without a valid state binding can still become a legacy Case v1, but it cannot claim candidate-generation coverage.

The logical executor identity freezes the candidate retrieval contract, `portable-postings-v2`, tokenizer, channel merge, temporal visibility, and token estimator. Source physical-index provenance records such fields as strategy, backend, ANN version, readiness, coverage, degradation, and fallback reason. These are different claims: the first is portable executable behavior; the second explains what production used.

## From governed evidence to Case v2

A Case can only be curated from the latest canonical Retrieval Adjudication for a Context when that adjudication is:

- `accepted`;
- authenticated and bound to a managed evaluator principal;
- current at the materialization cutoff; and
- eligible after effective Retrieval Adjudication Evidence Controls are applied.

The Runtime reconstructs this evidence server-side. A caller cannot supply arbitrary query text, state, labels, Memory values, evaluator identity, or control state to `POST /v1/memory-cases`.

When the Context has Replay State, `memory-case-v2` projects the captured state into Case-local sequential aliases (`m1`, `m2`, ...). The payload contains:

- query, budget, system time, and evaluation time;
- every captured Memory revision required by candidate generation;
- the observed candidate aliases and source channels;
- the observed policy, logical executor, and physical-index provenance;
- positive, hard-negative, ignore, missed-positive, and unjudged alias sets; and
- state, Context binding, index identity, scenario, Oracle, and payload integrity evidence.

An unjudged alias is a Memory present in the captured state without a positive, hard-negative, or ignore label. It is not assumed correct or incorrect. This distinction is required once a candidate policy can introduce Memories that production never put in front of the reviewer.

`case_id` remains a random UUID-derived `mcase_<32 hex>` token. Clients must treat it as opaque and must not infer content equality from it. Internal scenario and Oracle fingerprints let the Store converge equivalent evidence on one Case without deriving the public ID from content. Multiple eligible sources may attach to that Case; each remains an immutable `memory-case-source-v1` with its own provenance and evidence-control fingerprint.

Case summaries expose only opaque identity, contract/type, rubric, coverage, source and label counts, and timestamps. Full Case detail is `restricted_memory_content` and available only through `evaluation:export`. Stable state/content correlators do not appear in ordinary evaluation reads, Change Feed payloads, Causality facts, logs, or telemetry.

Memory Case creation retains the principal-scoped idempotency boundary `(Workspace, creator principal, SHA-256(Idempotency-Key))`. An exact retry returns the same Case/source result; changed input conflicts; the raw key is never stored. Workspace Snapshot v13 carries the hashed result ledger and state collections so retry and replay behavior survive restore.

## Running candidate-generation replay

Create a Run with 1-200 unique Case IDs from one scenario, a candidate Context Policy, an optional baseline Context Policy, and threshold overrides:

```http
POST /v1/replay-runs
Authorization: Bearer <evaluation:execute key>
Idempotency-Key: replay-context-v4
Content-Type: application/json

{
  "case_ids": ["mcase_0123456789abcdef0123456789abcdef"],
  "candidate_context_policy": {
    "name": "candidate-context",
    "version": 4,
    "weights": {
      "effective_confidence": 0.45,
      "lexical_overlap": 0.3,
      "freshness": 0.15,
      "semantic_similarity": 0.1
    },
    "freshness_decay_days": 60,
    "kind_boosts": { "constraint": 0.2, "identity": 0.1 },
    "filters": {
      "minimum_confidence": 0,
      "minimum_source_trust": 0,
      "blocked_kinds": []
    },
    "maximum_candidates": 1000,
    "candidate_pool": {
      "strategy": "portable_postings",
      "lexical_share": 0.7,
      "protected_share": 0.1,
      "recent_share": 0.2
    },
    "semantic_failure_mode": "fail_closed"
  },
  "thresholds": {
    "minimum_recall": 0.9,
    "maximum_hard_negative_hit_rate": 0,
    "maximum_unjudged_selected": 0
  }
}
```

The Runtime freezes the active Context Policy as baseline when one is not supplied. It normalizes thresholds and stores hashes for the Case set, both policies, and thresholds. One Run cannot mix Case v1 and Case v2 scenarios.

For each Case v2, the executor independently rebuilds baseline and candidate pools from the same state. Portable policies replay lexical, protected, and recent channels through the captured logical contract. A semantic policy embeds the query and captured Memory content through the configured provider, computes exhaustive scalar cosine similarity, then applies the same channel-budget and merge rules. This makes semantic model behavior testable without storing raw vectors, but it is not a byte-for-byte reconstruction of an approximate HNSW traversal.

Each per-Case result reports:

- generated candidate aliases, additions/removals from the observed pool, and channel metadata;
- selected/ineligible aliases, ranks, component scores, and contributions;
- positive misses, hard-negative hits, and candidate misses;
- unjudged selections and Oracle coverage;
- token budget and compliance; and
- stable `improved`, `unchanged`, or `regressed` classification and reasons.

The aggregate gate checks trusted evidence, Recall/Precision/MRR/nDCG, hard-negative rate, pairwise accuracy, candidate misses, budget compliance, no regression, and unjudged selection. `maximum_unjudged_selected` defaults to zero. This is intentionally conservative: a candidate policy must not pass by selecting unknown state that the Oracle never judged.

Execution remains durable and lease-fenced. Runs move through `queued -> running -> completed|failed|cancelled`; Case results are immutable; every transition is append-only. Heartbeats renew the ownership epoch, lease loss aborts execution, and a stale worker cannot commit results. `resume` reclaims queued or expired work without displacing a live owner. Workspace restore queues in-flight work with a higher fence.

## Production Write Quality Cohort Plans

A Replay Run remains the right unit for an ad hoc comparison of 1-200 Cases. A production Write Quality release can instead create `replay-cohort-plan-v1`. The Store atomically fixes the database cutoff, reconstructs the complete latest-authenticated same-rubric consensus population, pins governed member evidence, partitions the sorted Case set into shards, and creates the shard Runs. The request supplies rubric, candidate Memory Policy, optional baseline, thresholds, and shard size; it cannot supply Case IDs or the cutoff.

One Plan is bounded to 10,000 Cases, 100 shards, and 200 Cases per shard. Its derived state is `queued`, `running`, `partial`, `completed`, `failed`, or `cancelled`. Resume advances unfinished shard Runs without rewriting completed results; cancellation is durable. Ordinary Plan projections expose population counts, progress, opaque Run IDs, shard states, and integrity hashes. Store-only member and source evidence is available to Release construction and recovery, not to the Console or an `evaluation:read` caller.

A completed Plan can become stale when a new Case becomes eligible, a pinned Case is no longer eligible, or its latest authenticated semantic judgment changes. Memory Store Contract v19 therefore makes freshness a Store operation rather than a client-side list comparison. One atomic current cutoff covers the full same-rubric latest-authenticated-consensus population; the result classifies added, removed, changed, and unchanged Cases, then projects only counts and content hashes. Case IDs and current source/evaluator evidence do not leave the trusted Store path.

Operators can run `memplumb cohort-plan-check --plan rcp_... --maximum-age-seconds 604800 --json`. The equivalent HTTP endpoint is `GET /v1/replay-cohort-plans/{planId}/freshness?maximum_age_seconds=604800`; TypeScript and Python sync/async SDKs provide typed methods, and the bilingual Console shows the population comparison, evidence age, validity deadline, and gate. Seven days is the default maximum evidence age. A population change, blocked reconstruction, or expiry requires a new Plan rather than mutating the old one.

## What the coverage proves

The current coverage declaration remains exact:

| Layer                    | `memory_state_retrieval.v1` | Meaning                                                                  |
| ------------------------ | --------------------------- | ------------------------------------------------------------------------ |
| Candidate pool           | covered                     | Baseline and candidate pools are regenerated from the captured state.    |
| Logical candidate index  | covered                     | Versioned tokenizer/postings/channel/temporal contracts are replayed.    |
| Retrieval scoring/filter | covered                     | Context Policy evaluation runs independently for both policies.          |
| Context budget           | covered                     | Ranking and token selection are recomputed.                              |
| Semantic similarity      | covered                     | The bound provider recomputes scalar similarities over captured content. |
| Historical physical ANN  | not covered                 | HNSW graph/search execution is represented by provenance, not replayed.  |
| Extraction               | not covered                 | Source events are not re-extracted.                                      |
| Write policy             | not covered                 | Replay does not mutate or reconstruct Memory writes.                     |

`candidate_index=true` therefore means the portable logical index contract, not a claim that a PostgreSQL HNSW graph was serialized and rerun. Physical ANN readiness and recall remain release-qualified by `semantic-index-status` and the real Store `semantic-index-benchmark`.

`frozen_retrieval.v1` retains its original, narrower meaning: it evaluates scoring, filters, semantic similarity, and budget over captured candidate snapshots, with `candidate_pool=false` and `candidate_index=false`.

## Binding replay evidence to Release Artifacts

```powershell
memplumb release-create `
  --policy policies\candidate.json `
  --baseline-context-policy policies\context-default.json `
  --context-policy policies\context-candidate.json `
  --quality-replay-run-id rrun_... `
  --output release.json
```

Release artifact schema v9 accepts either supported retrieval Replay scope. A state-aware release binding requires a completed Run, the complete Case set, `split=all`, every completed immutable Case result, matching policy/threshold/dataset/result hashes, consistent coverage, and a passed gate. It additionally binds a privacy-safe state manifest and Case-result manifest containing State, Context binding, index identity, logical executor, and result hashes. The aggregate unjudged count is recomputed from persisted Case results before release approval. Independent `memory_write_counterfactual` and complete-population `production_write_quality_replay` gates can be attached without replacing production retrieval evidence.

Release Artifact v10 extends only the production Write Quality binding to a batch of 2-100 homogeneous, disjoint Runs while preserving the 200-Case limit on each Run. The earliest Run creation time is the cohort cutoff; the combined Case manifest must exactly equal the complete same-rubric eligible population at that cutoff, up to the bounded 10,000-Case release scan. MemPlumb revalidates each shard and then recomputes one global aggregate and gate. Artifact v9 remains the output for a single Write Quality Run and remains verifiable.

Release Artifact v11 introduced the Store-native completed Plan binding. Release Artifact v12 adds a fresh atomic current-population attestation and a maximum-evidence-age validity deadline when `release-create` receives `--write-quality-cohort-plan-id`. It verifies the frozen Plan/member/shard/current-Run graph and pinned evidence, recomputes the full-population aggregate, and fails closed when current membership changed, eligibility cannot be proven, or the Plan is too old. The output binds counts and manifests without exporting private Case payloads, freshness Case IDs, member eligibility snapshots, source records, Actor IDs, or evaluator details.

Restricted Case payloads and sources do not enter the release artifact. A failed or partial Replay Run, a non-ready Cohort Plan, or a failed freshness gate blocks release. Artifact v12 also fails closed at signing and trusted deployment after `valid_until`; long-running canary routing disables an expired candidate and records the transition before returning stable. Live rubric/cutoff replay remains a mutually exclusive compatibility path. Artifact v8-v11 verification remains supported.

## Operations, recovery, and lifecycle

Memory Store Contract v20 requires both built-in adapters to implement atomic candidate-plus-state capture, Context Replay State resolution, `createMemoryCaseFromCurrentSourceIdempotent`, all Case/Run scopes, Replay Cohort Plan creation/read/cancel/evidence/freshness, ingest and Replay lease fencing, Actor lifecycle, causality, and recovery. Retrieval Case creation re-reads and locks the current Context and Adjudication, re-materializes the restricted projection, and compares its complete content and Source inside the final write transaction. Actor purge or Workspace replacement therefore linearizes before a rejected create or after a create that the lifecycle operation removes. SQLite schema v26 and PostgreSQL migration v32 are the current physical contracts.

Actor export schema v14 includes Actor-owned State Snapshots/Context bindings, every Actor-bound ingest claim state without its claim-token hash, and linked retrieval/write Case, Run, and Cohort Plan evidence. Hard purge removes Actor-owned claims, state, and sources, then removes orphaned Cases and affected Plan/Run evidence. Workspace Snapshot v17 carries state, all Case source families, Write Adjudications, Runs, Cohort Plan/member/shard/attempt/event collections, review evidence, and the Causality Ledger, but intentionally excludes ingest claims. Physical HNSW projections remain derived and are rebuilt after restore. Release Artifact v12 is the current freshness-bound Store-native Cohort format; v8-v11 remain verifiable.

Generic trace-to-dataset replay is already a market baseline. MemPlumb's narrower specialization is the governed chain from exact production Memory state and authenticated judgment to candidate-generation counterfactual, conservative unjudged gate, durable result, and release evidence. Current v5/v6 write scopes cover Memory Policy behavior and authenticated exact-action quality while binding policy-evaluation time separately from Event time and preserving the observed Memory status/update boundary. Strict v3/v4 readers retain existing evidence without pretending to add those newer fields. None of these scopes pretends to replay extraction or final-state reduction. Answer-level evaluation, hosted onboarding, and broad framework integration remain future work rather than implied capabilities.

See [HTTP API](./http-api.md), [Retrieval quality loop](./retrieval-quality-loop.md), [Memory Write Counterfactual](./memory-write-counterfactual.md), [Production Memory Write Quality](./memory-write-quality.md), [Security](./security.md), [Store adapters](./store-adapters.md), and [Disaster recovery](./disaster-recovery.md).