# Workspace disaster recovery

MemPlumb Workspace Snapshot is a storage-neutral, content-addressed logical backup. It preserves the evidence required to reproduce memory behavior across SQLite and PostgreSQL:

- current memories and version history;
- raw events, pipeline traces, policy decisions, and review labels;
- Context Runs, every considered retrieval candidate and exact Memory revision, ranking evidence, feedback, Retrieval Adjudication revisions/exact labels/evidence controls, quality review tasks/assignments/events, Outcome Ledger evidence/controls, and embedding cache;
- content-addressed Memory State Snapshots and Context Replay State bindings captured with Context candidate generation;
- opaque-ID/content-deduplicated Memory Cases and source provenance, the principal-scoped hashed Case idempotency ledger, Replay Runs, per-Case results, and append-only Run events;
- Replay Cohort Plans, pinned member eligibility evidence, shard manifests, current Run attempts, and append-only Plan events;
- deletion receipts;
- API Key hashes, scopes, revocation state, and metadata;
- stable Change Feed IDs and payloads.

It excludes plaintext API tokens, raw quality-review or Replay lease tokens and their hashes, provider credentials, in-flight claims, raw idempotency keys, Outcome/Retrieval Adjudication/Replay idempotency metadata, exported `quality-dataset-v1` files, ephemeral rate-limit windows, managed Change Consumer leases/checkpoints, Change Feed retention floors, Webhook endpoints/encrypted secrets/delivery jobs, global rollout state, portable posting rows, semantic profile DDL, and physical HNSW projections. Snapshot v17 retains Memory Case idempotency hashes/references, restricted state collections, both write source families, Write Adjudications, and the complete Replay Cohort Plan graph so Memory Case exact retries, state-aware replays, authenticated Write Quality evidence, and durable shard progress survive recovery. Existing API tokens continue to authenticate after a complete restore because their hashes are snapshot data. Active review work is fenced, and a running Replay Run restores as `queued` with cleared lease material and a higher version; the restored Plan derives its progress from those current Runs. Restore also clears stale limiter, consumer, Webhook, and Workspace semantic-projection state. Webhook master keys, Embedding Provider credentials, raw lease tokens, and exported quality datasets require separate recovery procedures.

Snapshot v17 preserves the historical Plan graph, not a promise that it is still fresh after restore. Cohort freshness is intentionally derived from the restored Store's complete current latest-authenticated-consensus population. It is recomputed atomically under Memory Store Contract v20 and is not copied as mutable Plan state. A previously completed Plan can therefore restore correctly yet still be stale, blocked, or older than the release policy.

The MemPlumb v0.13.2 compatibility boundary is Memory Store Contract v20, SQLite schema v26, PostgreSQL migration v32, Workspace Snapshot v17, Actor export schema v14, and Release Artifact v12 with v8-v11 verification compatibility. A recovery target must migrate to its current physical schema before importing a v17 snapshot.

Export is point-in-time consistent. SQLite holds an immediate transaction while reading the Workspace. PostgreSQL uses one `REPEATABLE READ READ ONLY` transaction, so writes committed during export cannot split an event from its decisions or traces in the artifact.

## Create and sign

Use a dedicated backup signing key. The same Ed25519 format and Key ID convention used for trusted releases is supported.

```powershell
memplumb signing-keygen `
  --private-key .secrets/backup-private.pem `
  --public-key backup-public.pem

memplumb backup-create `
  --store postgres `
  --workspace support-prod `
  --output support-prod.snapshot.json `
  --private-key .secrets/backup-private.pem `
  --issuer production-backup

memplumb backup-verify `
  --file support-prod.snapshot.json `
  --public-key backup-public.pem `
  --json
```

The SHA-256 content hash detects corruption. The optional Ed25519 attestation proves that the snapshot came from a holder of the trusted private key. Production recovery should require the public key; an unsigned integrity check is not an authenticity check.

## Restore

Restore into an empty Workspace whenever possible:

```powershell
$env:MEMPLUMB_DATABASE_URL = "postgresql://memplumb:secret@recovery-db/memplumb"
memplumb schema-migrate --store postgres
memplumb backup-restore `
  --store postgres `
  --workspace support-prod `
  --file support-prod.snapshot.json `
  --public-key backup-public.pem
```

A non-empty target is rejected. `--force` deletes and replaces only the selected Workspace inside the same transaction. Any invalid reference, duplicate global ID, API Key hash collision, or SQL failure rolls back the entire restore.

Snapshot v2 includes Change Feed records. Restore assigns new Store-local cursors, appends `workspace.restored`, and clears named consumers and retention floors; downstream consumers must recreate their consumer, bootstrap from the restored history, and deduplicate preserved records by stable change ID. Snapshot v1 artifacts remain verifiable and restorable but contain no Change Feed collection.

Snapshot v3 adds Outcome Ledger evidence. Snapshot v4 adds Outcome Evidence Controls. Snapshot v5 adds Memory Causality; v6 complete Context candidates; v7 adjudications/labels; v8 retrieval controls; v9 legacy review tasks/events; v10 independent review entities; v11 Memory Cases and Replay Runs; v12 principal-scoped Case idempotency; v13 Memory State/Context Replay bindings; v14 Write Counterfactual sources; v15 Write Adjudications; v16 authenticated Write Quality sources; and v17 Replay Cohort Plans, members, shards, current attempts, and Plan events. Snapshot v17 is the current default. Active review leases are independently reset and fenced. Running Replay leases are converted to queued work with a higher fence, and Plan progress is recomputed from the restored shard Runs.

Snapshot v1-v16 artifacts remain verifiable and restorable, but cannot recover collections introduced later. In particular, v12 preserves Cases/Runs and deterministic Case retries but has no contemporaneous Memory State evidence; v15 has Write Adjudications but no Write Quality Source collection; and v16 has authenticated Write Quality sources but no Store-native Cohort Plan graph. Restored v1 Cases remain valid for `frozen_retrieval.v1`; they must not be upgraded to candidate-generation coverage by reconstructing state after the fact.

For a `hybrid_ann` deployment, restore preserves canonical embedding-cache rows but deliberately removes physical projection rows. A profile may still be provisioned at the target database level, yet the restored Workspace reports `building` until current Memory revision/content hashes are projected again. This prevents an apparently healthy index from serving a pre-restore or mismatched revision.

After restore:

1. Run `schema-status` and require `integrity: verified` with no pending migration.
2. Run `backup-create` against the restored Workspace and compare collection counts.
3. Inspect representative Event Traces and Context Runs, including `evaluated_at`, candidate ordinals, sources, eligibility, exclusions, selected ranks, adjudication revisions, exact labels, missed-positive resolution, retrieval control history, and task generation/event order.
4. Verify an existing API token and a read-only memory query.
5. Rematerialize a known rubric/cutoff quality dataset and compare its dataset fingerprint, evidence-control fingerprint, matching/quarantined counts, and conflict/evidence counts with the recovery record. Keep that restricted asset outside ordinary logs and build artifacts.
6. Compare Memory State Snapshot/Context binding, Memory Case/source, Replay Run/result/event, and Cohort Plan/member/shard/attempt/event counts. Verify representative State, binding, index identity, Case, Plan, shard, and result hashes; confirm a previously running shard Run is queued with cleared lease identity and a higher version, then resume the Plan and compare its completed manifests.
7. Run `cohort-plan-check --plan rcp_... --maximum-age-seconds 604800 --json` for every Plan intended for release. Require zero added, removed, and changed Cases plus a passing age check. The HTTP/SDK/Console view is also acceptable for operators, but its privacy projection intentionally omits Case IDs. Create and complete a new Plan when the restored evidence is stale, blocked, or expired; do not waive freshness because snapshot hashes match.
8. For `hybrid_ann`, ensure the exact provider/dimension profile is provisioned, run `semantic-index-backfill`, require `semantic-index-status` to meet `minimum_coverage`, and rerun `semantic-index-benchmark` on the target Store.
9. Reconcile quality review tasks and verify that database-clock-expired leases reject renew/complete and can be taken over with a higher `lease_version`.
10. Start the daemon and require `/ready` to return `200` before shifting traffic. A `portable_fallback` Runtime may be ready with `degraded: true`; treat that as a recovery exception, not full service restoration. Revalidate every Artifact v12 at deployment time; signing before backup or restore does not extend `valid_until`.

## Cross-store recovery

Snapshot JSON is deliberately independent of SQLite and PostgreSQL physical formats. A SQLite Workspace can restore to PostgreSQL and back without changing canonical IDs, Context candidate evidence, State Snapshot hashes/bindings, review/control revisions, Case provenance, Replay results, or Cohort Plan/shard relationships. Cross-Workspace restore rebuilds Workspace-bound Write Quality Observation, Adjudication, Source, member eligibility, Plan, and result integrity while preserving portable Case semantics. Portable term postings and physical HNSW tables are derived rather than copied. State-aware replay therefore reconstructs the versioned logical index from the portable state, while the target Store must separately backfill and qualify its physical ANN profile. The target adapter then performs a new atomic Cohort population check; freshness is never inferred from source-store success.

## RPO, RTO, and security

Snapshots are full logical backups. The recovery point objective equals the snapshot schedule; there is currently no incremental or WAL shipping mode. Recovery time grows with Workspace evidence volume because import validates and writes every record transactionally. For `hybrid_ann`, RTO also includes Embedding Provider throughput, backfill pages, HNSW projection writes, and the post-restore semantic gate. Capacity drills must measure that phase rather than stopping when relational import completes.

Snapshots contain personal data, retrieval explanations, candidate membership, adjudication metadata/evidence, Memory Case queries/values/source references, Cohort member eligibility snapshots, Case creator principals and hashed idempotency/request correlation, Replay policy and score evidence, task assignments and historical principal ownership, control reasons/operator principals, and derived vectors. They deliberately exclude raw idempotency/review/Replay lease tokens and Replay token hashes. Encrypt them at rest and in transit using the backup platform, restrict access separately from the runtime database, rotate signing keys, and apply a retention policy consistent with deletion and regulatory requirements. Snapshot deletion is required when its retained data exceeds the approved backup window; an application deletion receipt cannot erase an older external backup. The same limitation applies to exported quality datasets, whose query and Memory content make them more sensitive than the privacy-safe Release Artifact v12 Cohort summary. The public freshness attestation contains comparison counts and hashes, never the added, removed, changed, or unchanged Case-ID lists retained inside the Store calculation.

Run a restore drill on a schedule and record the snapshot ID, content hash, source/target adapters, counts, elapsed recovery time, and post-restore verification result. A backup that has never been restored is not accepted disaster-recovery evidence.