kfickel/streamd

Fork 0

feat(cache): local graph-database cache for localized shards (Grafeo) #142

New issue

Closed

opened 2026-09-06 15:07:53 +02:00 by kfickel · 3 comments

kfickel commented

2026-09-06 15:07:53 +02:00

Owner

Context

Every streamd invocation re-reads and re-parses the entire markdown corpus. load_markdown_shards (src/cli/commands/mod.rs:20) walks the base folder, fs::read_to_strings every .md file, parses it, and localizes it — for all of todo, edit, daily, and timesheet. The LSP does the same thing recursively, and compute_diagnostics re-walks the whole corpus on every diagnostic request.

On a local SSD this is free: measured on the personal corpus (126 files, 235 KB) streamd todo runs in 12 ms end-to-end including process startup. The real motivation is different: on the work WSL machine the stream directory lives on a OneDrive-synced share where file I/O is very expensive, and that corpus is already ~300 files. There the cost is dominated by read_to_string per file, not by parsing.

Goals, in order:

Avoid the file reads. Validate cached data with stat only (size + mtime); read from disk only the files that actually changed.
Enable graph queries streamd cannot express today — tag co-occurrence, dimension/project rollups, "all shards in dimension X" as a one-hop traversal rather than a full-tree scan.
Learning — work with a real embedded graph database in a real project.

Backend: Grafeo (pure Rust, embeddable, GQL/Cypher). Location: .streams-cache.grafeo inside the base folder. Graph model: shards + tags + dimension values as first-class nodes.

Why Grafeo

flake.nix builds four targets — native, musl-static (-C target-feature=+crt-static), mingw-windows cross, and the wasm Zed extension — with strictDeps = true and empty buildInputs: the project today has zero C dependencies, and the .deb is built from the musl static binary and executes it inside the Nix sandbox.

That constraint eliminates the strongest engine in the field: Kuzu was archived in Oct 2025 (Apple acquisition) and its live successor LadybugDB (lbug, 0.19.1) is a C++ core via cxx. Of the pure-Rust options, CozoDB is abandoned (no commits since 2024) and Oxigraph is pure-Rust only in-memory (on-disk = RocksDB). SurrealDB embedded is pure Rust with kv-surrealkv but drags in a dependency tree that would dwarf this 7.9k-line project. Grafeo fits every build constraint.

Grafeo is young — created Jan 2026, v0.5.42, pre-1.0 file format, largely one author. That is acceptable only because this database is pure derived state: it is never authoritative, and every failure path rebuilds it from the markdown. The design below makes that guarantee structural.

Known trade-off

.streams-cache.grafeo sits inside the OneDrive-synced folder, so OneDrive will upload it on every change (sync churn) and may create conflict copies when two machines write it. The design neutralises the correctness half of this — any unreadable, corrupt, or version-mismatched database is deleted and rebuilt — but the churn remains. Add .streams-cache.grafeo* to .gitignore and exclude it from OneDrive sync. Switching to ProjectDirs::cache_dir() later is a one-line change, since the path is produced by a single function.

Design

Key insight: split the graph into two layers

localize_stream_file is pure — its only inputs are (Shard tree, file name, RepositoryConfiguration, Tz), with no I/O and no clock. That lets the cache be split so the correctness-critical path never depends on which configuration was used:

Structural layer (:File, :Shard, HAS_CHILD) — config- and timezone-independent. This is what replaces the expensive file reads.
Projection layer (:Tag, :DimensionValue, TAGGED, PLACED_IN) — materialized under one canonical config + tz, purely additive, used only for the new graph queries.

This matters because the same folder is loaded under different configurations today: todo/edit use TaskConfiguration, timesheet uses BasicTimesheetConfiguration, the LSP uses the merge of both; and todo/edit hardcode chrono_tz::UTC while timesheet uses the repo timezone. Caching LocalizedShards directly would force the config and tz into the cache key.

Instead, the read path reconstructs the raw Shard tree from the structural layer and calls the existing localize_stream_file with the caller's own config and tz — in memory, no I/O.

Consequence: zero behaviour change. Every command produces byte-identical output to today, including the existing todo/edit UTC quirk.

Graph model

Nodes:

Label	Properties
`:Meta`	`schema_version`, `streamd_version`, `projection_fingerprint` (singleton)
`:File`	`path`, `size`, `mtime_ns`, `parse_ok`
`:Shard`	`start_line`, `end_line`, `ordinal`, `markers` (list), `tags` (list)
`:Tag`	`name` (globally deduplicated)
`:DimensionValue`	`dimension`, `value` (globally deduplicated)

Edges:

(:File)-[:HAS_ROOT]->(:Shard)
(:Shard)-[:HAS_CHILD]->(:Shard) — ordinal on the child preserves Vec order
(:Shard)-[:IN_FILE]->(:File) — denormalized; makes subgraph deletion and file lookup one hop
(:Shard)-[:TAGGED]->(:Tag)
(:Shard)-[:PLACED_IN]->(:DimensionValue) — carries an order property

PLACED_IN stores the full effective location (including entries inherited from ancestors), so "all shards in project X" is a single hop. The file dimension is excluded — it is already IN_FILE, and duplicating a path string onto every node would bloat the graph for nothing. Scale check: ~300 files × ~10 shards × ~4 dimensions ≈ 12k edges. Trivial.

markers/tags are order-significant Vec<String>, so they are stored as Value::List, not as edges. (Tag nodes are the query projection; the list property is the reconstruction source.)

Invalidation

Whole database: Meta.schema_version mismatch, Meta missing, or any error opening or reading the database ⇒ remove the cache path (file or directory, see store.rs below) and rebuild from scratch.
Projection only: projection_fingerprint (stable hash of the canonical merged RepositoryConfiguration serialized via the existing serde_json dependency, plus the tz name) mismatch ⇒ drop and rebuild :Tag/:DimensionValue and their edges, keep the structural layer.
Per file: WalkDir + entry.metadata() — stat only, no reads. Compare (size, mtime_ns) against the :File node. Changed or new ⇒ read + parse + insert. Present in graph but gone from disk ⇒ delete its subgraph via IN_FILE.

Files that fail to localize are recorded with parse_ok = false so they are not re-read on every run; today load_markdown_shards silently drops them.

The scan is recursive (matching the LSP's list_markdown_files, skipping dot-directories). load_markdown_shards_cached then filters to depth 1 to preserve the CLI's current max_depth(1) behaviour exactly. This existing CLI/LSP inconsistency is preserved deliberately, not fixed silently.

Concurrency

Grafeo's GrafeoDB::open_read_only takes a shared file lock, which implies read-write takes an exclusive one — so a long-lived handle in the LSP would lock out every CLI invocation.

Therefore: all database access is short-lived — open, work, close. No handle outlives a single operation. The LSP keeps its existing DashMap for the per-keystroke path and touches the database only at startup (the cold-start warm-up, which is exactly the OneDrive pain point) and on watched-file changes.

On any lock-acquisition failure, or any Grafeo error at all, fall back to the existing uncached load_markdown_shards. A user command must never fail because of the cache.

Implementation

1. Dependency (`Cargo.toml`)

grafeo = { version = "0.5", default-features = false, features = ["gql"] }

Enable exactly one query language. Do not enable jemalloc or mimalloc-allocator — both are C libraries and would break the zero-C-dependency build across musl-static and mingw.

2. New module `src/cache/`

mod.rs — public API, re-exports
schema.rs — label/edge/property name constants, SCHEMA_VERSION, projection_fingerprint()
store.rs — path resolution (base_folder.join(".streams-cache.grafeo")), open/close, AccessMode/lock handling, rebuild_on_error wrapper.

The .grafeo extension is deliberate. Per the Config docs, StorageFormat::Auto "detects the format from the path: .grafeo extension uses single-file format, directories use the legacy WAL directory" — so .grafeo gets the single-file format with no explicit with_storage_format call and no direct grafeo-engine dependency. (StorageFormat is not re-exported from the grafeo crate; it lives at grafeo_engine::config::StorageFormat, which would otherwise have to be version-pinned in lockstep.)

Defensively, clear() and the rebuild path must still remove both a file and a directory at the cache path — fs::remove_file, falling back to fs::remove_dir_all, treating "already absent" as success — since the format is extension-driven in a pre-1.0 crate and could change.
sync.rs — stat scan, diff against :File nodes, apply inserts/deletes in one transaction
read.rs — reconstruct Vec<StreamFile> from the structural layer
query.rs — graph-native queries (new capability)

Public surface:

pub fn load_markdown_shards_cached(
    base_folder: &Path,
    config: &RepositoryConfiguration,
    tz: Tz,
) -> Result<Vec<LocalizedShard>, StreamdError>;

pub fn rebuild(base_folder: &Path) -> Result<CacheStats, StreamdError>;
pub fn clear(base_folder: &Path) -> Result<(), StreamdError>;
pub fn status(base_folder: &Path) -> Result<CacheStatus, StreamdError>;

load_markdown_shards_cached is a drop-in replacement for load_markdown_shards: sync the structural layer, reconstruct StreamFiles, then map through the existing localize_stream_file(&stream_file, config, tz) (src/localize/shard.rs), dropping errors exactly as the current loader does.

Use Grafeo's direct-access API (create_node_with_props, create_edge, get_node, get_neighbors_outgoing) for the hot sync/reconstruct paths — the docs put these at 10–30× the speed of an equivalent MATCH. Reserve GQL for query.rs.

3. Wire in the call sites

src/cli/commands/mod.rs — keep load_markdown_shards as the uncached fallback; add load_markdown_shards_cached and switch todo.rs, edit.rs, daily.rs, timesheet.rs to it. Signatures are unchanged, so the four commands need a one-line edit each.
src/cli/commands/lsp.rs — in LspState construction, seed file_cache from the database instead of parsing from disk; on did_change_watched_files, update the changed file's subgraph. Leave parse_and_cache and the DashMap hot path untouched.
src/cli/args.rs + new src/cli/commands/cache.rs — streamd cache rebuild|clear|status.
Honour a STREAMD_NO_CACHE=1 escape hatch (clap already has the env feature enabled).

4. Errors (`src/error.rs`)

Add CacheError(String) and a SerdeJsonError(#[from] serde_json::Error) variant — StreamdError currently has no JSON variant. Cache errors must be internal: they trigger fallback, they do not propagate to the user.

5. New graph queries (`src/cache/query.rs`)

The capability this buys, expressed as GQL over the projection layer:

tags_co_occurring_with(tag) — shards tagged X, then out to their other :Tag nodes
shards_in_dimension(dimension, value) — one hop via PLACED_IN, replacing the recursive find_shard_by_position full-tree scan (src/query/find.rs)
dimension_value_counts(dimension) — project/task-state rollups
files_touching_date(date) — the index compute_diagnostics recomputes from scratch on every request today

Keep src/query/find.rs untouched and working on Vec<LocalizedShard>; these are additive.

6. Docs

Per CLAUDE.md, update both:

REQUIREMENTS.md — new section after R25 covering the cache file location, the stat-based invalidation contract, the schema-version/rebuild guarantee, the fallback-on-any-error rule, and the cache subcommand. Amend R25 to note the LSP's cold-start seeding.
README.md — the streamd cache command, the .streams-cache.grafeo file, and the .gitignore / OneDrive-exclusion recommendation.

Verification

Unit/integration tests (cargo test), using the existing tempfile dev-dependency:

Round-trip fidelity — parse a corpus, insert, reconstruct, assert the reconstructed Vec<LocalizedShard> == the directly-computed one. Run it under TaskConfiguration, BasicTimesheetConfiguration, and the merge, plus two timezones, to prove config-independence.
Invalidation — build cache; touch one file's content; assert only that file is re-read and the result matches a cold parse.
Deletion — remove a file, assert its subgraph is gone and it vanishes from results.
Corruption recovery — write garbage bytes over .streams-cache.grafeo, assert the command still succeeds and the database is rebuilt.
Schema bump — write a :Meta node with an old schema_version, assert full rebuild.
Lock contention — hold an exclusive handle, assert load_markdown_shards_cached falls back and returns correct results.

End-to-end:

nix develop
cargo test && cargo clippy && cargo fmt --check

# Real corpus, correctness first: output must be identical with and without the cache
cd "$STREAM_DIR"
STREAMD_NO_CACHE=1 streamd todo > /tmp/uncached.txt
streamd todo > /tmp/cached.txt && diff /tmp/uncached.txt /tmp/cached.txt
STREAMD_NO_CACHE=1 streamd timesheet > /tmp/ts-uncached.txt
streamd timesheet > /tmp/ts-cached.txt && diff /tmp/ts-uncached.txt /tmp/ts-cached.txt

# Then measure: cold (first run, builds cache) vs warm
streamd cache clear && time streamd todo && time streamd todo

Build constraints — the part most likely to break, so check all targets:

nix flake check
nix build .#streamd-musl       # zero-C-dependency static build must still link
nix build .#streamd-windows    # mingw cross must still work
nix build .#streamd-deb        # runs the musl binary inside the sandbox

The honest success criterion is the warm run on the WSL/OneDrive machine, since that is the only place the current cost is real — locally the baseline is already 12 ms. Capture a before/after number there before considering this done.

Revision: cache file renamed from .streams-cache.db to .streams-cache.grafeo — see the refinement comment below for the reasoning.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

## Context Every streamd invocation re-reads and re-parses the entire markdown corpus. `load_markdown_shards` (`src/cli/commands/mod.rs:20`) walks the base folder, `fs::read_to_string`s every `.md` file, parses it, and localizes it — for all of `todo`, `edit`, `daily`, and `timesheet`. The LSP does the same thing recursively, and `compute_diagnostics` re-walks the whole corpus on every diagnostic request. On a local SSD this is free: measured on the personal corpus (126 files, 235 KB) `streamd todo` runs in **12 ms end-to-end including process startup**. The real motivation is different: on the work WSL machine the stream directory lives on a **OneDrive-synced share where file I/O is very expensive**, and that corpus is already ~300 files. There the cost is dominated by `read_to_string` per file, not by parsing. Goals, in order: 1. **Avoid the file reads.** Validate cached data with `stat` only (size + mtime); read from disk only the files that actually changed. 2. **Enable graph queries** streamd cannot express today — tag co-occurrence, dimension/project rollups, "all shards in dimension X" as a one-hop traversal rather than a full-tree scan. 3. **Learning** — work with a real embedded graph database in a real project. Backend: **Grafeo** (pure Rust, embeddable, GQL/Cypher). Location: `.streams-cache.grafeo` inside the base folder. Graph model: shards + tags + dimension values as first-class nodes. ### Why Grafeo `flake.nix` builds four targets — native, **musl-static** (`-C target-feature=+crt-static`), **mingw-windows cross**, and the wasm Zed extension — with `strictDeps = true` and **empty `buildInputs`: the project today has zero C dependencies**, and the `.deb` is built from the musl static binary and *executes* it inside the Nix sandbox. That constraint eliminates the strongest engine in the field: Kuzu was archived in Oct 2025 (Apple acquisition) and its live successor LadybugDB (`lbug`, 0.19.1) is a C++ core via `cxx`. Of the pure-Rust options, CozoDB is abandoned (no commits since 2024) and Oxigraph is pure-Rust only in-memory (on-disk = RocksDB). SurrealDB embedded is pure Rust with `kv-surrealkv` but drags in a dependency tree that would dwarf this 7.9k-line project. Grafeo fits every build constraint. Grafeo is young — created Jan 2026, v0.5.42, pre-1.0 file format, largely one author. That is acceptable **only** because this database is pure derived state: it is never authoritative, and every failure path rebuilds it from the markdown. The design below makes that guarantee structural. ### Known trade-off `.streams-cache.grafeo` sits inside the OneDrive-synced folder, so OneDrive will upload it on every change (sync churn) and may create conflict copies when two machines write it. The design neutralises the *correctness* half of this — any unreadable, corrupt, or version-mismatched database is deleted and rebuilt — but the churn remains. Add `.streams-cache.grafeo*` to `.gitignore` and exclude it from OneDrive sync. Switching to `ProjectDirs::cache_dir()` later is a one-line change, since the path is produced by a single function. ## Design ### Key insight: split the graph into two layers `localize_stream_file` is **pure** — its only inputs are `(Shard tree, file name, RepositoryConfiguration, Tz)`, with no I/O and no clock. That lets the cache be split so the correctness-critical path never depends on which configuration was used: - **Structural layer** (`:File`, `:Shard`, `HAS_CHILD`) — config- and timezone-independent. This is what replaces the expensive file reads. - **Projection layer** (`:Tag`, `:DimensionValue`, `TAGGED`, `PLACED_IN`) — materialized under one canonical config + tz, purely additive, used only for the new graph queries. This matters because the same folder is loaded under **different configurations** today: `todo`/`edit` use `TaskConfiguration`, `timesheet` uses `BasicTimesheetConfiguration`, the LSP uses the merge of both; and `todo`/`edit` hardcode `chrono_tz::UTC` while `timesheet` uses the repo timezone. Caching `LocalizedShard`s directly would force the config and tz into the cache key. Instead, the read path reconstructs the raw `Shard` tree from the structural layer and calls the existing `localize_stream_file` with **the caller's own config and tz** — in memory, no I/O. **Consequence: zero behaviour change.** Every command produces byte-identical output to today, including the existing `todo`/`edit` UTC quirk. ### Graph model Nodes: | Label | Properties | |---|---| | `:Meta` | `schema_version`, `streamd_version`, `projection_fingerprint` (singleton) | | `:File` | `path`, `size`, `mtime_ns`, `parse_ok` | | `:Shard` | `start_line`, `end_line`, `ordinal`, `markers` (list), `tags` (list) | | `:Tag` | `name` (globally deduplicated) | | `:DimensionValue` | `dimension`, `value` (globally deduplicated) | Edges: - `(:File)-[:HAS_ROOT]->(:Shard)` - `(:Shard)-[:HAS_CHILD]->(:Shard)` — `ordinal` on the child preserves `Vec` order - `(:Shard)-[:IN_FILE]->(:File)` — denormalized; makes subgraph deletion and file lookup one hop - `(:Shard)-[:TAGGED]->(:Tag)` - `(:Shard)-[:PLACED_IN]->(:DimensionValue)` — carries an `order` property `PLACED_IN` stores the **full effective** `location` (including entries inherited from ancestors), so "all shards in project X" is a single hop. The `file` dimension is *excluded* — it is already `IN_FILE`, and duplicating a path string onto every node would bloat the graph for nothing. Scale check: ~300 files × ~10 shards × ~4 dimensions ≈ 12k edges. Trivial. `markers`/`tags` are order-significant `Vec<String>`, so they are stored as `Value::List`, not as edges. (`Tag` nodes are the query projection; the list property is the reconstruction source.) ### Invalidation - **Whole database**: `Meta.schema_version` mismatch, `Meta` missing, or *any* error opening or reading the database ⇒ remove the cache path (file *or* directory, see `store.rs` below) and rebuild from scratch. - **Projection only**: `projection_fingerprint` (stable hash of the canonical merged `RepositoryConfiguration` serialized via the existing `serde_json` dependency, plus the tz name) mismatch ⇒ drop and rebuild `:Tag`/`:DimensionValue` and their edges, keep the structural layer. - **Per file**: `WalkDir` + `entry.metadata()` — **stat only, no reads**. Compare `(size, mtime_ns)` against the `:File` node. Changed or new ⇒ read + parse + insert. Present in graph but gone from disk ⇒ delete its subgraph via `IN_FILE`. Files that fail to localize are recorded with `parse_ok = false` so they are not re-read on every run; today `load_markdown_shards` silently drops them. The scan is **recursive** (matching the LSP's `list_markdown_files`, skipping dot-directories). `load_markdown_shards_cached` then filters to depth 1 to preserve the CLI's current `max_depth(1)` behaviour exactly. This existing CLI/LSP inconsistency is preserved deliberately, not fixed silently. ### Concurrency Grafeo's `GrafeoDB::open_read_only` takes a *shared* file lock, which implies read-write takes an exclusive one — so a long-lived handle in the LSP would lock out every CLI invocation. Therefore: **all database access is short-lived — open, work, close.** No handle outlives a single operation. The LSP keeps its existing `DashMap` for the per-keystroke path and touches the database only at startup (the cold-start warm-up, which is exactly the OneDrive pain point) and on watched-file changes. **On any lock-acquisition failure, or any Grafeo error at all, fall back to the existing uncached `load_markdown_shards`.** A user command must never fail because of the cache. ## Implementation ### 1. Dependency (`Cargo.toml`) ```toml grafeo = { version = "0.5", default-features = false, features = ["gql"] } ``` Enable exactly one query language. **Do not enable `jemalloc` or `mimalloc-allocator`** — both are C libraries and would break the zero-C-dependency build across musl-static and mingw. ### 2. New module `src/cache/` - `mod.rs` — public API, re-exports - `schema.rs` — label/edge/property name constants, `SCHEMA_VERSION`, `projection_fingerprint()` - `store.rs` — path resolution (`base_folder.join(".streams-cache.grafeo")`), open/close, `AccessMode`/lock handling, `rebuild_on_error` wrapper. The `.grafeo` extension is deliberate. Per the `Config` docs, `StorageFormat::Auto` "detects the format from the path: `.grafeo` extension uses single-file format, directories use the legacy WAL directory" — so `.grafeo` gets the single-file format with **no** explicit `with_storage_format` call and **no** direct `grafeo-engine` dependency. (`StorageFormat` is not re-exported from the `grafeo` crate; it lives at `grafeo_engine::config::StorageFormat`, which would otherwise have to be version-pinned in lockstep.) Defensively, `clear()` and the rebuild path must still remove **both** a file and a directory at the cache path — `fs::remove_file`, falling back to `fs::remove_dir_all`, treating "already absent" as success — since the format is extension-driven in a pre-1.0 crate and could change. - `sync.rs` — stat scan, diff against `:File` nodes, apply inserts/deletes in one transaction - `read.rs` — reconstruct `Vec<StreamFile>` from the structural layer - `query.rs` — graph-native queries (new capability) Public surface: ```rust pub fn load_markdown_shards_cached( base_folder: &Path, config: &RepositoryConfiguration, tz: Tz, ) -> Result<Vec<LocalizedShard>, StreamdError>; pub fn rebuild(base_folder: &Path) -> Result<CacheStats, StreamdError>; pub fn clear(base_folder: &Path) -> Result<(), StreamdError>; pub fn status(base_folder: &Path) -> Result<CacheStatus, StreamdError>; ``` `load_markdown_shards_cached` is a drop-in replacement for `load_markdown_shards`: sync the structural layer, reconstruct `StreamFile`s, then map through the **existing** `localize_stream_file(&stream_file, config, tz)` (`src/localize/shard.rs`), dropping errors exactly as the current loader does. Use Grafeo's direct-access API (`create_node_with_props`, `create_edge`, `get_node`, `get_neighbors_outgoing`) for the hot sync/reconstruct paths — the docs put these at 10–30× the speed of an equivalent `MATCH`. Reserve GQL for `query.rs`. ### 3. Wire in the call sites - `src/cli/commands/mod.rs` — keep `load_markdown_shards` as the uncached fallback; add `load_markdown_shards_cached` and switch `todo.rs`, `edit.rs`, `daily.rs`, `timesheet.rs` to it. Signatures are unchanged, so the four commands need a one-line edit each. - `src/cli/commands/lsp.rs` — in `LspState` construction, seed `file_cache` from the database instead of parsing from disk; on `did_change_watched_files`, update the changed file's subgraph. Leave `parse_and_cache` and the `DashMap` hot path untouched. - `src/cli/args.rs` + new `src/cli/commands/cache.rs` — `streamd cache rebuild|clear|status`. - Honour a `STREAMD_NO_CACHE=1` escape hatch (clap already has the `env` feature enabled). ### 4. Errors (`src/error.rs`) Add `CacheError(String)` and a `SerdeJsonError(#[from] serde_json::Error)` variant — `StreamdError` currently has no JSON variant. Cache errors must be *internal*: they trigger fallback, they do not propagate to the user. ### 5. New graph queries (`src/cache/query.rs`) The capability this buys, expressed as GQL over the projection layer: - `tags_co_occurring_with(tag)` — shards tagged X, then out to their other `:Tag` nodes - `shards_in_dimension(dimension, value)` — one hop via `PLACED_IN`, replacing the recursive `find_shard_by_position` full-tree scan (`src/query/find.rs`) - `dimension_value_counts(dimension)` — project/task-state rollups - `files_touching_date(date)` — the index `compute_diagnostics` recomputes from scratch on every request today Keep `src/query/find.rs` untouched and working on `Vec<LocalizedShard>`; these are additive. ### 6. Docs Per `CLAUDE.md`, update both: - `REQUIREMENTS.md` — new section after R25 covering the cache file location, the stat-based invalidation contract, the schema-version/rebuild guarantee, the fallback-on-any-error rule, and the `cache` subcommand. Amend R25 to note the LSP's cold-start seeding. - `README.md` — the `streamd cache` command, the `.streams-cache.grafeo` file, and the `.gitignore` / OneDrive-exclusion recommendation. ## Verification Unit/integration tests (`cargo test`), using the existing `tempfile` dev-dependency: 1. **Round-trip fidelity** — parse a corpus, insert, reconstruct, assert the reconstructed `Vec<LocalizedShard>` `==` the directly-computed one. Run it under `TaskConfiguration`, `BasicTimesheetConfiguration`, and the merge, plus two timezones, to prove config-independence. 2. **Invalidation** — build cache; touch one file's content; assert only that file is re-read and the result matches a cold parse. 3. **Deletion** — remove a file, assert its subgraph is gone and it vanishes from results. 4. **Corruption recovery** — write garbage bytes over `.streams-cache.grafeo`, assert the command still succeeds and the database is rebuilt. 5. **Schema bump** — write a `:Meta` node with an old `schema_version`, assert full rebuild. 6. **Lock contention** — hold an exclusive handle, assert `load_markdown_shards_cached` falls back and returns correct results. End-to-end: ```bash nix develop cargo test && cargo clippy && cargo fmt --check # Real corpus, correctness first: output must be identical with and without the cache cd "$STREAM_DIR" STREAMD_NO_CACHE=1 streamd todo > /tmp/uncached.txt streamd todo > /tmp/cached.txt && diff /tmp/uncached.txt /tmp/cached.txt STREAMD_NO_CACHE=1 streamd timesheet > /tmp/ts-uncached.txt streamd timesheet > /tmp/ts-cached.txt && diff /tmp/ts-uncached.txt /tmp/ts-cached.txt # Then measure: cold (first run, builds cache) vs warm streamd cache clear && time streamd todo && time streamd todo ``` Build constraints — the part most likely to break, so check all targets: ```bash nix flake check nix build .#streamd-musl # zero-C-dependency static build must still link nix build .#streamd-windows # mingw cross must still work nix build .#streamd-deb # runs the musl binary inside the sandbox ``` The honest success criterion is the **warm run on the WSL/OneDrive machine**, since that is the only place the current cost is real — locally the baseline is already 12 ms. Capture a before/after number there before considering this done. --- *Revision: cache file renamed from `.streams-cache.db` to `.streams-cache.grafeo` — see the refinement comment below for the reasoning.* https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel added the

planned

label

2026-09-06 15:07:53 +02:00

kfickel commented

2026-09-06 15:10:05 +02:00

Author

Owner

Refinement: resolve the storage-format open question

The plan body left one item explicitly unverified — "Grafeo's StorageFormat::Auto infers from the file extension and may not recognise .db; also confirm whether Config::persistent takes a file or a directory." That is now answered, and the answer changes a decision.

Finding

Per the Config docs, the storage_format field behaves as:

"Auto" (default) detects the format from the path: ".grafeo" extension uses single-file format, directories use the legacy WAL directory.

So Config::persistent(path) accepts either a file or a directory, and Auto picks the format from the path shape:

Path	Format under `Auto`
`*.grafeo`	single-file
directory	legacy WAL directory
`*.db` (our case)	not specified by the docs

.streams-cache.db matches neither documented case. The realistic outcomes are that it is treated as the legacy WAL directory format — i.e. Grafeo creates a directory named .streams-cache.db — or that it errors. Either way the plan's clear() and corruption-recovery paths, which assume a single file to fs::remove_file, would be wrong.

Second finding: StorageFormat is not re-exported from the grafeo crate. Its path is grafeo_engine::config::StorageFormat, and grafeo's public item list (structs, enums, traits at 0.5.42) does not include it. Calling with_storage_format explicitly therefore means adding a direct grafeo-engine dependency and pinning it in lockstep with grafeo — for a pre-1.0 crate, an ongoing version-coupling cost.

Recommended change

Rename the cache file to .streams-cache.grafeo.

This takes the documented single-file path, needs no explicit with_storage_format call, avoids the second-crate dependency on grafeo-engine, and keeps clear() a plain fs::remove_file. It still satisfies the original requirement — a dot-prefixed cache file inside the base folder — since only the extension differs from .streams-cache.db.

Consequent edits to the plan:

§2 store.rs: path becomes base_folder.join(".streams-cache.grafeo"); drop the with_storage_format note and the "file or directory" TODO.
§6 Docs and the Context trade-off note: .gitignore entry becomes .streams-cache.grafeo*, and likewise for the OneDrive exclusion.
Verification test 4 (corruption recovery): unchanged in intent, but write garbage over .streams-cache.grafeo.

Alternative if the .db extension is preferred: add grafeo-engine as a direct dependency, pass .with_storage_format(StorageFormat::SingleFile) explicitly (confirm the exact variant name at implementation time), and pin both crates to the same version.

Defensive requirement either way

Grafeo is pre-1.0 and the format is extension-driven, so a future version could change what a given path produces. clear() and the rebuild-on-error path must handle both a file and a directory at the cache path — try fs::remove_file, fall back to fs::remove_dir_all, and treat "already absent" as success. This keeps the plan's core guarantee intact: any unreadable or unexpected on-disk state is discarded and rebuilt from the markdown, never surfaced to the user.

Unchanged

Everything else in the plan stands: the two-layer structural/projection split, stat-only invalidation, short-lived open→work→close database access with fallback to load_markdown_shards on any error, and the build-target verification matrix.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

## Refinement: resolve the storage-format open question The plan body left one item explicitly unverified — *"Grafeo's `StorageFormat::Auto` infers from the file extension and may not recognise `.db`; also confirm whether `Config::persistent` takes a file or a directory."* That is now answered, and the answer changes a decision. ### Finding Per the `Config` docs, the `storage_format` field behaves as: > "Auto" (default) detects the format from the path: ".grafeo" extension uses single-file format, directories use the legacy WAL directory. So `Config::persistent(path)` accepts **either** a file or a directory, and `Auto` picks the format from the path shape: | Path | Format under `Auto` | |---|---| | `*.grafeo` | single-file | | directory | legacy WAL directory | | `*.db` (our case) | **not specified by the docs** | `.streams-cache.db` matches neither documented case. The realistic outcomes are that it is treated as the legacy WAL directory format — i.e. Grafeo creates a **directory** named `.streams-cache.db` — or that it errors. Either way the plan's `clear()` and corruption-recovery paths, which assume a single file to `fs::remove_file`, would be wrong. Second finding: `StorageFormat` is **not re-exported from the `grafeo` crate**. Its path is `grafeo_engine::config::StorageFormat`, and `grafeo`'s public item list (structs, enums, traits at 0.5.42) does not include it. Calling `with_storage_format` explicitly therefore means adding a direct `grafeo-engine` dependency and pinning it in lockstep with `grafeo` — for a pre-1.0 crate, an ongoing version-coupling cost. ### Recommended change **Rename the cache file to `.streams-cache.grafeo`.** This takes the documented single-file path, needs no explicit `with_storage_format` call, avoids the second-crate dependency on `grafeo-engine`, and keeps `clear()` a plain `fs::remove_file`. It still satisfies the original requirement — a dot-prefixed cache file inside the base folder — since only the extension differs from `.streams-cache.db`. Consequent edits to the plan: - §2 `store.rs`: path becomes `base_folder.join(".streams-cache.grafeo")`; drop the `with_storage_format` note and the "file or directory" TODO. - §6 Docs and the Context trade-off note: `.gitignore` entry becomes `.streams-cache.grafeo*`, and likewise for the OneDrive exclusion. - Verification test 4 (corruption recovery): unchanged in intent, but write garbage over `.streams-cache.grafeo`. **Alternative if the `.db` extension is preferred:** add `grafeo-engine` as a direct dependency, pass `.with_storage_format(StorageFormat::SingleFile)` explicitly (confirm the exact variant name at implementation time), and pin both crates to the same version. ### Defensive requirement either way Grafeo is pre-1.0 and the format is extension-driven, so a future version could change what a given path produces. `clear()` and the rebuild-on-error path must handle **both** a file and a directory at the cache path — try `fs::remove_file`, fall back to `fs::remove_dir_all`, and treat "already absent" as success. This keeps the plan's core guarantee intact: any unreadable or unexpected on-disk state is discarded and rebuilt from the markdown, never surfaced to the user. ### Unchanged Everything else in the plan stands: the two-layer structural/projection split, stat-only invalidation, short-lived open→work→close database access with fallback to `load_markdown_shards` on any error, and the build-target verification matrix. https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel commented

2026-09-06 15:16:42 +02:00

Author

Owner

Decision: accepted. The cache file is .streams-cache.grafeo.

The issue description has been updated in place, so it is now the single authoritative plan — the implementer does not need to reconcile it against the comment above. Changes applied:

Cache path is base_folder.join(".streams-cache.grafeo") throughout (Context, store.rs, docs section, corruption-recovery test).
store.rs no longer calls with_storage_format, and no grafeo-engine dependency is added — StorageFormat::Auto maps .grafeo to the single-file format on its own.
clear() and the rebuild path must remove both a file and a directory at the cache path, as a hedge against the extension-driven format changing in this pre-1.0 crate.
.gitignore / OneDrive exclusion pattern is now .streams-cache.grafeo*.

The planned label remains attached; no other part of the design changed.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

**Decision: accepted.** The cache file is `.streams-cache.grafeo`. The issue description has been updated in place, so it is now the single authoritative plan — the implementer does not need to reconcile it against the comment above. Changes applied: - Cache path is `base_folder.join(".streams-cache.grafeo")` throughout (Context, `store.rs`, docs section, corruption-recovery test). - `store.rs` no longer calls `with_storage_format`, and no `grafeo-engine` dependency is added — `StorageFormat::Auto` maps `.grafeo` to the single-file format on its own. - `clear()` and the rebuild path must remove both a file and a directory at the cache path, as a hedge against the extension-driven format changing in this pre-1.0 crate. - `.gitignore` / OneDrive exclusion pattern is now `.streams-cache.grafeo*`. The `planned` label remains attached; no other part of the design changed. https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel referenced this issue

2026-09-06 18:14:29 +02:00

feat(cache): local graph-database cache for localized shards (Grafeo) #143

kfickel commented

2026-09-06 18:16:21 +02:00

Author

Owner

Implemented — PR #143

Cost


Wall clock, first to last commit	155 min
...of which idle	~137 min (a single gap between 15:36 and 17:53)
Active implementation	~20 min across two sittings
Research + Grafeo API spike, before first commit	~20 min
Final verification (4 Nix build targets)	~14 min
Commits	10
Diff	23 files, +2499 / -23
New tests	50 (211 → 261)

Token usage: not reported. The session's context was compacted partway through, so the running counter reset and I cannot read a trustworthy cumulative figure. Rather than estimate one, the measurable proxies are above.

Findings

The plan's dependency line was wrong. grafeo = { default-features = false, features = ["gql"] } enables neither the LPG property-graph store nor persistence — gql maps only to grafeo-engine/gql, and GrafeoDB::open is gated behind wal. Nothing would have compiled against it. Correct set is ["edge", "storage"]. Worth noting the failure mode: a trivial spike (fn main(){}) compiled fine against the wrong features, so this only surfaced by reading the feature graph in the vendored source.

Grafeo held up. 46 crates, no *-sys/cc/bindgen, ~10 s clean build. The zero-C-dependency premise survived all four targets, including the musl static-pie binary and the mingw cross. The .grafeo extension does produce a single file, as the refinement predicted.

The exclusive lock is stricter than assumed. A held read-write handle blocks open_read_only too, not just other writers. The short-lived open→work→close discipline and the fallback path were therefore load-bearing, not defensive extras.

Two real bugs, both caught by tests rather than review:

Lock contention destroyed the cache. open_or_rebuild treated any open failure as corruption and deleted the file, so a second process would wipe a database the first was actively using. Now distinguished and propagated untouched; pinned by test_a_locked_cache_is_not_deleted.
load_markdown_shards was order-nondeterministic. It returned WalkDir order, and consumers use a stable sort_by_key(|s| s.moment) — so two files sharing a timestamp could order arbitrarily between runs. Both paths now sort on the timestamp-prefixed filename.

Also: GQL literals needed escaping, since the tag pattern admits apostrophes (@don't).

A test premise was wrong, not the code. No preconfigured marker populates the project dimension — it is declared but awaits repo-level config. My first rollup test assumed @Project-X did something; retargeted at task and file_type, which are actually populated.

The honest result

On this machine the cache is a net loss: 14 ms warm vs 13 ms uncached, 25 ms cold. Parsing 235 KB is already trivial, so graph open + stat + reconstruct costs marginally more than it saves. Correctness is solid — todo and timesheet are byte-identical with and without it — but the performance premise remains unproven until it runs against the OneDrive share. If it does not pay off there, STREAMD_NO_CACHE=1 and streamd cache clear make it inert.

Deviations

:Day nodes added. files_touching_date needs dates, which are localization output rather than structural facts; modelling days as nodes keeps canonical moments out of the config-independent layer.
No incremental LSP subgraph update. Redundant given stat-based invalidation — the next process to open the cache re-reads just the changed file.

Needs your action

Your stream folder is a git repo with no ignore entry for the cache. Add .streams-cache.grafeo* to its .gitignore, and exclude the file from OneDrive sync — it is a binary rewritten on every note change, so syncing it causes upload churn and cross-machine conflict copies. I left no cache file behind there.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

## Implemented — PR #143 ### Cost | | | |---|---| | Wall clock, first to last commit | 155 min | | ...of which idle | ~137 min (a single gap between 15:36 and 17:53) | | Active implementation | ~20 min across two sittings | | Research + Grafeo API spike, before first commit | ~20 min | | Final verification (4 Nix build targets) | ~14 min | | Commits | 10 | | Diff | 23 files, +2499 / -23 | | New tests | 50 (211 → 261) | **Token usage: not reported.** The session's context was compacted partway through, so the running counter reset and I cannot read a trustworthy cumulative figure. Rather than estimate one, the measurable proxies are above. ### Findings **The plan's dependency line was wrong.** `grafeo = { default-features = false, features = ["gql"] }` enables neither the LPG property-graph store nor persistence — `gql` maps only to `grafeo-engine/gql`, and `GrafeoDB::open` is gated behind `wal`. Nothing would have compiled against it. Correct set is `["edge", "storage"]`. Worth noting the failure mode: a trivial spike (`fn main(){}`) compiled fine against the wrong features, so this only surfaced by reading the feature graph in the vendored source. **Grafeo held up.** 46 crates, no `*-sys`/`cc`/`bindgen`, ~10 s clean build. The zero-C-dependency premise survived all four targets, including the musl static-pie binary and the mingw cross. The `.grafeo` extension does produce a single file, as the refinement predicted. **The exclusive lock is stricter than assumed.** A held read-write handle blocks `open_read_only` too, not just other writers. The short-lived open→work→close discipline and the fallback path were therefore load-bearing, not defensive extras. **Two real bugs, both caught by tests rather than review:** 1. *Lock contention destroyed the cache.* `open_or_rebuild` treated any open failure as corruption and deleted the file, so a second process would wipe a database the first was actively using. Now distinguished and propagated untouched; pinned by `test_a_locked_cache_is_not_deleted`. 2. *`load_markdown_shards` was order-nondeterministic.* It returned `WalkDir` order, and consumers use a *stable* `sort_by_key(|s| s.moment)` — so two files sharing a timestamp could order arbitrarily between runs. Both paths now sort on the timestamp-prefixed filename. Also: GQL literals needed escaping, since the tag pattern admits apostrophes (`@don't`). **A test premise was wrong, not the code.** No preconfigured marker populates the `project` dimension — it is declared but awaits repo-level config. My first rollup test assumed `@Project-X` did something; retargeted at `task` and `file_type`, which are actually populated. ### The honest result **On this machine the cache is a net loss:** 14 ms warm vs 13 ms uncached, 25 ms cold. Parsing 235 KB is already trivial, so graph open + stat + reconstruct costs marginally more than it saves. Correctness is solid — `todo` and `timesheet` are byte-identical with and without it — but the *performance* premise remains unproven until it runs against the OneDrive share. If it does not pay off there, `STREAMD_NO_CACHE=1` and `streamd cache clear` make it inert. ### Deviations - **`:Day` nodes added.** `files_touching_date` needs dates, which are localization output rather than structural facts; modelling days as nodes keeps canonical moments out of the config-independent layer. - **No incremental LSP subgraph update.** Redundant given stat-based invalidation — the next process to open the cache re-reads just the changed file. ### Needs your action Your stream folder is a git repo with no ignore entry for the cache. Add `.streams-cache.grafeo*` to its `.gitignore`, and exclude the file from OneDrive sync — it is a binary rewritten on every note change, so syncing it causes upload churn and cross-machine conflict copies. I left no cache file behind there. https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel closed this issue

2026-09-12 17:26:51 +02:00

No labels

planned

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

kfickel/streamd#142

No description provided.

Rows
Columns

feat(cache): local graph-database cache for localized shards (Grafeo) #142

Context

Why Grafeo

Known trade-off

Design

Key insight: split the graph into two layers

Graph model

Invalidation

Concurrency

Implementation

1. Dependency (Cargo.toml)

2. New module src/cache/

3. Wire in the call sites

4. Errors (src/error.rs)

5. New graph queries (src/cache/query.rs)

6. Docs

Verification

Refinement: resolve the storage-format open question

Finding

Recommended change

Defensive requirement either way

Unchanged

Implemented — PR #143

Cost

Findings

The honest result

Deviations

Needs your action

1. Dependency (`Cargo.toml`)

2. New module `src/cache/`

4. Errors (`src/error.rs`)

5. New graph queries (`src/cache/query.rs`)