> ## Documentation Index
> Fetch the complete documentation index at: https://docs.deyta.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Dream phase

> Offline maintenance pass for accumulated knowledge: entity dedupe, schema-drift audits, and graph hygiene, with a plan/apply split and undo snapshots.

Dream is an **offline maintenance pass** for accumulated knowledge. You run it against a single namespace, usually on a schedule. It audits the graph, plans consolidation work, and (in `mode="apply"`) executes that plan with bi-temporal soft-delete and a per-op undo snapshot.

The name follows the complementary learning systems framework from neuroscience: ingestion is the fast, episodic path (`Khora.remember`), and dream is the slow, reorganizing path. Same store, different access pattern. See [Research and prior art](#research-and-prior-art) for the lineage.

<Warning>
  Dream phase is **experimental** and **opt-in**. It does nothing until you set `DreamConfig.enabled=True`, and every destructive op is off by default. Apply mode mutates the relational store (PostgreSQL) and mirrors the changes to the graph store post-commit. Read [Caveats](#caveats) before scheduling it in production.
</Warning>

## Why run it

Reach for dream when one of these is true:

* **Duplicate entities from independent ingest batches.** Near-duplicate variants like `"OpenAI"` / `"Open AI"` or `"Marie Skłodowska-Curie"` / `"M. Curie"` that the ingest-time resolver didn't merge. The dedupe op does a second, namespace-wide pass with all accumulated evidence.
* **Stale provenance after heavy `forget()` / `forget_session()`.** Entity and relationship `source_chunk_ids` keep pointing at deleted chunks, because the forget cascade removes the chunks but not the back-pointers. That leaves provenance inaccurate and can skew entity scoring. The GC op drops the dead references.
* **You changed your `ExpertiseConfig` and want to know whether the data still matches.** The schema-drift report diffs the entity and relationship types present in the data against your config and flags what diverged. It reports the drift. It does not re-extract old documents.
* **You want a reviewable "state of the graph" snapshot** before authorizing anything destructive. Dry-run produces exactly that.

## How to run it

The master switch is `DreamConfig.enabled` (env var `KHORA_DREAM_ENABLED`), default `False`. Per-op flags are off by default too. Turn on only what you need.

```python theme={null}
from khora import Khora, KhoraConfig, DreamConfig

kb = Khora(
    KhoraConfig(
        dream=DreamConfig(
            enabled=True,
            report_file_sink_enabled=True,
            report_collector_sink_enabled=True,
        ),
    ),
)

result = await kb.dream(namespace_id, mode="dry-run")

# result.ops is the aggregate counter, one OpSummary per op kind.
for summary in result.ops:
    print(f"{summary.op_type}: planned={summary.planned} "
          f"skipped={summary.skipped} failed={summary.failed}")
```

The signature:

```python theme={null}
async def dream(
    namespace: str | UUID,
    *,
    mode: str = "dry-run",                # "dry-run" | "apply"
    scope: DreamScope | None = None,
    ops: Iterable[OpKind] | None = None,
    config: DreamConfig | None = None,
    on_progress: Callable[[DreamProgress], None] | None = None,
    resume_from: UUID | None = None,
) -> DreamResult
```

**Always dry-run first, then apply.** Apply only performs what the dry-run reported. If ingest may have continued in between, re-run the dry-run right before applying and confirm `result.metadata["plan_hash"]` is unchanged. Planning is deterministic, so a changed hash means the data moved under you.

Run-level history is in `khora_dream_runs`, readable via `kb.dream_history(namespace)` whether or not a report sink is on.

## Caveats

* **Some apply ops are PostgreSQL-only.** Most vectorcypher mutation handlers run on both the Postgres and embedded `sqlite_lance` stacks. Three that bind raw UUIDs are gated to PostgreSQL: `centroid_recompute`, `source_chunk_ids_gc`, and `contradiction_reconcile`. On any other dialect the orchestrator's dialect gate reports those ops as `skipped` (raising `DreamBackendUnsupported` internally) rather than letting a driver error leak. `dedupe_entities` and `prune_edges` run on `sqlite_lance`.
* **The graph store is kept in sync.** When the namespace has a graph backend, apply mirrors each soft-delete or endpoint rewrite to the graph post-commit: Neo4j and Memgraph get an eventual-consistency mirror, so graph recall stays consistent with PostgreSQL. A mirror failure after the PG commit is recorded (counter `khora.dream.graph_mirror.partial_failure`) and queued in `khora_dream_runs.graph_mirror_pending` for the reconciler to retry. (`sqlite_lance` is single-store, so no mirror is needed.)
* **Undo is per-op, not per-run.** Every apply handler snapshots pre-state into `undo.json` (schema `dream-undo/1`). `kb.dream_undo(op_id)` reverses a single applied op from that snapshot inside one transaction. There is no run-level `undo(run_id)`, so reversing a whole run means undoing its ops one at a time.
* **Guardrails on the apply path.** A `KHORA_DREAM_DISABLE_APPLY` kill-switch, a Postgres advisory lock held for the whole run, a `chunk_id`-mutation runtime assertion, and the snapshot-before-mutate undo records.
* **Concurrency.** One run per namespace at a time (advisory lock); a second concurrent run fast-fails with `DreamLockUnavailable`. Different namespaces run in parallel. On embedded backends the lock degrades to an in-process `asyncio.Lock`, so cross-process safety is **not** promised on `sqlite_lance`.
* **It is not autonomous.** Dream does not decide *when* to run (that's your cron / Temporal / k8s policy), does not judge whether a planned merge makes business sense (it uses cosine / Levenshtein / age heuristics), and does not replace good ingest-time decisions. If dedupe finds thousands of merges in a fresh namespace, the bug is upstream.

**Not yet implemented:** `centroid_recompute` on `sqlite_lance`, and auto-chaining of dedupe into centroid recompute (today centroid recompute needs clusters fed in from a prior dedupe run).

## Phases and what they do

Every op returns a `DreamOp` with a `decision` string and a structured `outputs` dict. The orchestrator routes those through whichever sinks are enabled. An op never mutates state directly. Even Phase 2 planners only *describe* what they would do until you call apply.

### Phase 1: audit ops (read-only)

Pure observation. No LLM calls, no mutations, no risk to production data. Apply mode is a pass-through.

* **Schema drift vs `ExpertiseConfig`.** A multiset diff of the `entity_type` / `relationship_type` strings in the data against what your config declares: types present but undeclared, types declared but unused, and frequencies that shifted by 50% or more since the last run. It never renames anything.
* **PageRank orphan report.** Builds the namespace's entity-relationship graph, down-weights `ASSOCIATED_WITH` co-occurrence edges so they don't dominate, runs PageRank, and flags entities that are bottom-percentile, barely mentioned (`mention_count <= 1`), and not recently recalled. Each is marked `archive_candidate=true`. The op never archives.
* **`source_chunk_ids` array-length audit.** Reports dead-UUID counts, the array-length distribution, and the worst offenders. Surfaces GC candidates for the Phase 2 GC op without touching a row.

### Phase 2: planner ops

Each emits one `DreamOp` per work item. In `dry-run` you get the plan only. In `apply` the matching `apply_<op>` handler runs under a per-op transaction, snapshotting pre-state into `undo.json` first. The plan is checkpointed to `khora_dream_runs`, so a crashed run resumes via `resume_from=<run_id>`.

* **Cross-batch entity dedupe.** Buckets entities by `(name_lower, entity_type)`, scores pairwise cosine on pre-normalized embeddings, and plans a merge for any pair above the per-type threshold (default `0.90`, tighter than the online resolver's `0.85`). Skip-collisions, where one canonical entity would absorb two clusters, are reported but not auto-resolved.
* **Centroid recompute.** For each proposed merge cluster, decides how the canonical embedding should be produced: a weighted-mean `centroid` for close surface-form variants, a `re_embed` of the canonical name for lexically distant but semantically aligned names, or a `skip_multimodal` finding when the cluster spans more than one concept (the merge itself is the bug). Needs `rapidfuzz` (the `[accel]` extra).
* **`source_chunk_ids` GC.** Plans per-entity rewrites that drop the dead chunk UUIDs the audit found. Idempotent.

Four more planner ops ship behind their own enable flags, all off by default, so they don't run unless you opt in: edge pruning, contradiction detection across facts, community summaries (an LLM-using op), and schema-type normalization from an operator-supplied mapping. They're newer than the ops above, so dry-run them first.

The ops described above, by op-kind string:

| Op                                    | Phase       | Apply behavior                                                        |
| ------------------------------------- | ----------- | --------------------------------------------------------------------- |
| `vectorcypher_schema_drift_report`    | 1 (audit)   | pass-through                                                          |
| `vectorcypher_orphan_report`          | 1 (audit)   | pass-through                                                          |
| `vectorcypher_source_chunk_ids_audit` | 1 (audit)   | pass-through                                                          |
| `vectorcypher_dedupe_entities`        | 2 (planner) | bi-temporal soft-delete + relationship rewrite, mirrored to the graph |
| `vectorcypher_centroid_recompute`     | 2 (planner) | overwrites canonical entity embedding (Postgres-only)                 |
| `vectorcypher_source_chunk_ids_gc`    | 2 (planner) | array filter, dialect-aware, idempotent (Postgres-only apply)         |

## Reading the reports

Three sinks consume the same `DreamOp` stream. Enable each independently via `DreamConfig.report_*_sink_enabled`.

### File sink

Writes per-run artifacts under `{base_dir}/{namespace_id}/{date}/{run_id}.*`:

| File            | Contents                                                                          |
| --------------- | --------------------------------------------------------------------------------- |
| `summary.md`    | Human-readable summary plus sampled high-impact ops. Start here.                  |
| `events.jsonl`  | One `DreamOp` per line, machine-readable, schema `dream-report/1`                 |
| `manifest.json` | Run metadata and checksum                                                         |
| `undo.json`     | Pre-state snapshots per mutating op (apply mode); empty for audit-only or dry-run |

`redact_text` (`"none"` / `"summary"` / `"all"`, default `"summary"`) controls how much raw text appears across all sinks. Old reports are swept by `retention_days` (default 30) and `retention_runs_per_namespace` (default 50).

**Default location.** With `report_file_sink_enabled=True`, reports write to `<system temp dir>/khora-dream-reports` (`/tmp/...` on Linux), which the OS can wipe on reboot. There is no config setting for the path, so copy reports somewhere persistent on a schedule if you need a durable audit trail.

### Event sink

Bridges into the existing `HookDispatcher` via `DREAM_*` event types (`DREAM_RUN_STARTED`, `DREAM_PHASE_STARTED`, `DREAM_OP_DECIDED`, `DREAM_PHASE_COMPLETED`, `DREAM_RUN_COMPLETED`, `DREAM_RUN_FAILED`). Existing `SemanticFilter` filters work, including the low-cost fields `dream_op_types` and `dream_decisions`. A `DREAM_OP_DECIDED` payload matches one line of `events.jsonl`.

### Collector sink (OpenTelemetry)

Emits spans and metrics. The operator-facing pieces are stable; the per-op internals may change, so don't pin dashboards to them.

* **Public spans:** `khora.dream.run`, `khora.dream.phase`.
* **Public metrics** (aggregate-only, never labeled by `namespace_id`): `khora.dream.runs_total`, `khora.dream.run.duration`, `khora.dream.phase.duration`, `khora.dream.ops_total {phase, op_type, decision}`.

Free-text attributes (rationale strings, entity names) are hashed before they become span attributes. Raw text is never a label.

## Reference

### Configuration

`DreamConfig` is a `pydantic-settings` model (env prefix `KHORA_DREAM_`). The knobs you'll reach for:

| Field                                                   | Default                 | Notes                                                            |
| ------------------------------------------------------- | ----------------------- | ---------------------------------------------------------------- |
| `enabled`                                               | `False`                 | Master switch; `kb.dream()` raises `DreamDisabledError` when off |
| `default_mode`                                          | `"dry-run"`             | Used when the caller omits `mode=`                               |
| `ops.{dedupe_entities,prune_edges,recompute_centroids}` | `False`                 | Per-op enable flags                                              |
| `report_{file,event,collector}_sink_enabled`            | `False`                 | Sink toggles                                                     |
| `redact_text`                                           | `"summary"`             | `"none"` / `"summary"` / `"all"`                                 |
| `retention_days` / `retention_runs_per_namespace`       | `30` / `50`             | Report retention                                                 |
| `dedupe_entities_default_threshold`                     | `0.90`                  | Cosine merge threshold                                           |
| `dedupe_entities_per_type_thresholds`                   | `{}`                    | Per-type overrides, e.g. `{"PERSON": 0.95}`                      |
| `source_chunk_ids_gc_min_dead`                          | `1`                     | Min dead-UUID count to plan a GC for an entity                   |
| `orphan_pr_percentile_threshold`                        | `5.0`                   | Bottom-percentile cut-off for the orphan report                  |
| `llm_max_tokens_per_run` / `_per_namespace_per_day`     | `200_000` / `1_000_000` | Token budgets (Phase 5)                                          |

Full env-var bindings are in [Configuration](/khora/configuration).

### Public API

Bound methods on `khora.Khora`, with functional equivalents at `khora.dream.api`:

* `dream(namespace, *, mode, scope, ops, config, on_progress, resume_from) -> DreamResult`
* `dream_status(run_id) -> dict`
* `dream_history(namespace, *, limit=20) -> list[DreamRunInfo]`
* `dream_undo(op_id, *, base_dir=None) -> bool`: reverse one applied op from its `undo.json` snapshot. Returns `True` when a row was restored, `False` for an unknown or already-undone op.

Public result types (re-exported from `khora`): `DreamResult`, `DreamRunInfo`, `DreamScope`, `DreamMode`, `OpKind`, `UndoRecord`, `OpSummary`. `DreamResult.metadata` carries `plan_hash` and (on dry-run) `plan_payload`.

### Exceptions

All inherit from `KhoraError`. Pattern-match these from a job runner:

| Exception               | When                                                                                                       |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| `DreamDisabledError`    | `DreamConfig.enabled` is `False`                                                                           |
| `DreamApplyDisabled`    | `KHORA_DREAM_DISABLE_APPLY` kill-switch tripped; apply refused without touching the DB                     |
| `DreamForbiddenOpError` | Plan contains a forbidden op (document delete, `chunk_id` mutation, UNIQUE collision, read-only namespace) |
| `DreamRunStuckError`    | A prior run is `applying` with a stale heartbeat; resolve via `resume_from` after review                   |
| `DreamLockUnavailable`  | Advisory-lock contention with a concurrent run on the same namespace                                       |

### Storage

* **`khora_dream_runs`** (PostgreSQL-only): the checkpoint table for crash-resume. Carries `state`, `plan_hash`, `last_committed_op_seq`, heartbeat timestamps, and the report path. The embedded path mirrors equivalent state through the file sink.
* **Bi-temporal columns** on `relationships` and `memory_facts`: `valid_to`, `invalidated_at`, `invalidated_by`. They record when a row was superseded and by which op, so a soft-delete can answer "what did the agent believe on date X" without losing history.

### Stability

| Symbol                                                                             | Tag                   |
| ---------------------------------------------------------------------------------- | --------------------- |
| `Khora.dream()`, `dream_status()`, `dream_history()`, `dream_undo()`               | public                |
| `DreamConfig`, `DreamResult`, `DreamRunInfo`, `DreamScope`, `OpKind`, `UndoRecord` | public                |
| Dream exceptions                                                                   | public                |
| Public OTel spans + aggregate metrics                                              | public                |
| `OpKind` *values*, per-op spans, planner/orchestrator internals                    | internal (may evolve) |

## Research and prior art

Dream phase is not a novel invention. It composes patterns the systems and ML communities have used for decades, applied to long-lived agent memory.

The naming comes from the complementary learning systems framework (McClelland, McNaughton & O'Reilly, 1995): fast episodic encoding and slow structured consolidation need separate substrates to avoid catastrophic interference. That maps onto online ingest versus offline replay. The same "ingest in one regime, consolidate in another" split shows up in offline RL replay buffers (DQN, prioritized experience replay) and in the OLTP-versus-OLAP separation, which is the cleanest framing: dream is to memory what OLAP is to OLTP.

The entity-resolution and bi-temporal pieces sit in established literature (Köpcke & Rahm 2010 on entity matching; Snodgrass 1999 on bi-temporal SQL). Compared with agent-memory frameworks, dream targets the *consolidation* side they each defer:

| System       | Does well                                | Gap dream addresses                                           |
| ------------ | ---------------------------------------- | ------------------------------------------------------------- |
| MemGPT       | OS-style paged recall vs. archival tiers | No structural audit of the archival tier over time            |
| GraphRAG     | Community-summary index built at ingest  | Re-indexing is a full rebuild, no incremental drift detection |
| Self-RAG     | Retrieval-on-demand with reflection      | Online only, no offline corpus hygiene                        |
| Letta / Mem0 | Structured user-facing memory blocks     | No scheduled compaction or dedupe pass                        |

It is complementary to all four: same substrate, different cadence, different objective.