Dream phase

​

Why run it

• Duplicate entities from independent ingest batches. Near-duplicate variants like "OpenAI" / "Open AI" or "Marie Skłodowska-Curie" / "M. Curie" that the ingest-time resolver didn’t merge. The dedupe op does a second, namespace-wide pass with all accumulated evidence.
• Stale provenance after heavy forget() / forget_session(). Entity and relationship source_chunk_ids keep pointing at deleted chunks, because the forget cascade removes the chunks but not the back-pointers. That leaves provenance inaccurate and can skew entity scoring. The GC op drops the dead references.
• You changed your ExpertiseConfig and want to know whether the data still matches. The schema-drift report diffs the entity and relationship types present in the data against your config and flags what diverged. It reports the drift. It does not re-extract old documents.
• You want a reviewable “state of the graph” snapshot before authorizing anything destructive. Dry-run produces exactly that.

​

How to run it

from khora import Khora, KhoraConfig, DreamConfig

kb = Khora(
    KhoraConfig(
        dream=DreamConfig(
            enabled=True,
            report_file_sink_enabled=True,
            report_collector_sink_enabled=True,
        ),
    ),
)

result = await kb.dream(namespace_id, mode="dry-run")

# result.ops is the aggregate counter, one OpSummary per op kind.
for summary in result.ops:
    print(f"{summary.op_type}: planned={summary.planned} "
          f"skipped={summary.skipped} failed={summary.failed}")

async def dream(
    namespace: str | UUID,
    *,
    mode: str = "dry-run",                # "dry-run" | "apply"
    scope: DreamScope | None = None,
    ops: Iterable[OpKind] | None = None,
    config: DreamConfig | None = None,
    on_progress: Callable[[DreamProgress], None] | None = None,
    resume_from: UUID | None = None,
) -> DreamResult

​

Caveats

• Some apply ops are PostgreSQL-only. Most vectorcypher mutation handlers run on both the Postgres and embedded sqlite_lance stacks. Three that bind raw UUIDs are gated to PostgreSQL: centroid_recompute, source_chunk_ids_gc, and contradiction_reconcile. On any other dialect the orchestrator’s dialect gate reports those ops as skipped (raising DreamBackendUnsupported internally) rather than letting a driver error leak. dedupe_entities and prune_edges run on sqlite_lance.
• The graph store is kept in sync. When the namespace has a graph backend, apply mirrors each soft-delete or endpoint rewrite to the graph post-commit: Neo4j and Memgraph get an eventual-consistency mirror, so graph recall stays consistent with PostgreSQL. A mirror failure after the PG commit is recorded (counter khora.dream.graph_mirror.partial_failure) and queued in khora_dream_runs.graph_mirror_pending for the reconciler to retry. (sqlite_lance is single-store, so no mirror is needed.)
• Undo is per-op, not per-run. Every apply handler snapshots pre-state into undo.json (schema dream-undo/1). kb.dream_undo(op_id) reverses a single applied op from that snapshot inside one transaction. There is no run-level undo(run_id), so reversing a whole run means undoing its ops one at a time.
• Guardrails on the apply path. A KHORA_DREAM_DISABLE_APPLY kill-switch, a Postgres advisory lock held for the whole run, a chunk_id-mutation runtime assertion, and the snapshot-before-mutate undo records.
• Concurrency. One run per namespace at a time (advisory lock); a second concurrent run fast-fails with DreamLockUnavailable. Different namespaces run in parallel. On embedded backends the lock degrades to an in-process asyncio.Lock, so cross-process safety is not promised on sqlite_lance.
• It is not autonomous. Dream does not decide when to run (that’s your cron / Temporal / k8s policy), does not judge whether a planned merge makes business sense (it uses cosine / Levenshtein / age heuristics), and does not replace good ingest-time decisions. If dedupe finds thousands of merges in a fresh namespace, the bug is upstream.

​

Phases and what they do

​

Phase 1: audit ops (read-only)

• Schema drift vs ExpertiseConfig. A multiset diff of the entity_type / relationship_type strings in the data against what your config declares: types present but undeclared, types declared but unused, and frequencies that shifted by 50% or more since the last run. It never renames anything.
• PageRank orphan report. Builds the namespace’s entity-relationship graph, down-weights ASSOCIATED_WITH co-occurrence edges so they don’t dominate, runs PageRank, and flags entities that are bottom-percentile, barely mentioned (mention_count <= 1), and not recently recalled. Each is marked archive_candidate=true. The op never archives.
• source_chunk_ids array-length audit. Reports dead-UUID counts, the array-length distribution, and the worst offenders. Surfaces GC candidates for the Phase 2 GC op without touching a row.

​

Phase 2: planner ops

• Cross-batch entity dedupe. Buckets entities by (name_lower, entity_type), scores pairwise cosine on pre-normalized embeddings, and plans a merge for any pair above the per-type threshold (default 0.90, tighter than the online resolver’s 0.85). Skip-collisions, where one canonical entity would absorb two clusters, are reported but not auto-resolved.
• Centroid recompute. For each proposed merge cluster, decides how the canonical embedding should be produced: a weighted-mean centroid for close surface-form variants, a re_embed of the canonical name for lexically distant but semantically aligned names, or a skip_multimodal finding when the cluster spans more than one concept (the merge itself is the bug). Needs rapidfuzz (the [accel] extra).
• source_chunk_ids GC. Plans per-entity rewrites that drop the dead chunk UUIDs the audit found. Idempotent.

​

Reading the reports

​

File sink

​

Event sink

​

Collector sink (OpenTelemetry)

• Public spans: khora.dream.run, khora.dream.phase.
• Public metrics (aggregate-only, never labeled by namespace_id): khora.dream.runs_total, khora.dream.run.duration, khora.dream.phase.duration, khora.dream.ops_total {phase, op_type, decision}.

​

Reference

​

Configuration

​

Public API

• dream(namespace, *, mode, scope, ops, config, on_progress, resume_from) -> DreamResult
• dream_status(run_id) -> dict
• dream_history(namespace, *, limit=20) -> list[DreamRunInfo]
• dream_undo(op_id, *, base_dir=None) -> bool: reverse one applied op from its undo.json snapshot. Returns True when a row was restored, False for an unknown or already-undone op.

​

Exceptions

​

Storage

• khora_dream_runs (PostgreSQL-only): the checkpoint table for crash-resume. Carries state, plan_hash, last_committed_op_seq, heartbeat timestamps, and the report path. The embedded path mirrors equivalent state through the file sink.
• Bi-temporal columns on relationships and memory_facts: valid_to, invalidated_at, invalidated_by. They record when a row was superseded and by which op, so a soft-delete can answer “what did the agent believe on date X” without losing history.

​

Stability

​

Research and prior art