feat(cache): local graph-database cache for localized shards (Grafeo) #143

Merged

kfickel merged 11 commits from 142_graph-database-cache into main 2026-09-06 21:08:09 +02:00

Conversation 1 Commits 11 Files changed 22 +2673 -23

kfickel commented 2026-09-06 18:14:29 +02:00

Owner

Copy link

Implements #142.

Caches parsed stream files in an embedded Grafeo graph at .streams-cache.grafeo inside the stream folder. Commands validate it with stat alone and re-read only files that actually changed.

Design

The graph is split into two layers, which is what makes the change safe:

• Structural (:File, :Shard, HAS_ROOT/HAS_CHILD/IN_FILE) — stores raw Shard trees. Independent of any RepositoryConfiguration or timezone.
• Projection (:Tag, :DimensionValue, :Day, TAGGED/PLACED_IN/ON_DAY) — derived by localizing under the canonical merged configuration. Purely additive; only graph queries read it.

Commands read the structural layer and re-run the pure localize_stream_file with their own configuration and timezone. That matters because the same folder is loaded three different ways today (todo/edit use TaskConfiguration at UTC, timesheet uses BasicTimesheetConfiguration at the repo timezone, the LSP merges both). Caching LocalizedShards directly would have forced config and tz into the cache key; this way output is unchanged.

Everything is derived state: a corrupt cache, an unknown schema version, or a lock held by another process all resolve to rebuilding or falling back. A cache failure can never fail a command.

Verified

On the real 126-file corpus, todo and timesheet are byte-identical with and without the cache. All four build targets pass:

Target	Result
nix flake check	passed
streamd-musl	static-pie linked, stripped
streamd-windows	PE32+ executable
streamd-deb	package created

261 tests pass (up from 211 on main — 50 new), including equivalence across both configurations × two timezones on cold and warm caches, corruption recovery, schema bump, deletion, and lock contention.

Performance: no local win

Measured on this machine (126 files, 235 KB, local SSD):

	time
uncached	13 ms
cold (builds cache)	25 ms
warm	14 ms

The warm path is 1 ms slower than not caching at all. Parsing 235 KB is already trivial, so graph open + stat + reconstruct costs slightly more than it saves. This is consistent with the issue's premise — the target is the WSL/OneDrive share where per-file reads dominate — but it means the benefit is still unproven. The honest success criterion remains a before/after warm run on the work machine; if it does not pay off there, STREAMD_NO_CACHE=1 and streamd cache clear make it a no-op.

Deviations from the plan

Three, each with a reason:

1. Feature set is ["edge", "storage"], not ["gql"]. The planned line enables neither the LPG property-graph store nor persistence — GrafeoDB::open does not exist under it. Dependency tree is 46 crates with no *-sys/cc/bindgen; neither allocator feature is enabled.
2. No incremental subgraph update on didSave/didChangeWatchedFiles. Redundant: invalidation is stat-based, so the next process to open the cache re-reads just that file. The graph is self-correcting.
3. :Day nodes added. The plan listed files_touching_date, but dates are localization output, not structural facts. Modelling days as nodes keeps canonical moments out of the config-independent layer while still making date questions a traversal.

Notable fixes found while building

• Lock contention destroyed the cache. open_or_rebuild treated any open failure as corruption and deleted the file — so a second process would wipe a database the first was actively using. Lock contention is now distinguished and propagated untouched. Pinned by test_a_locked_cache_is_not_deleted.
• load_markdown_shards was order-nondeterministic. It returned WalkDir order; since consumers use a stable sort_by_key(|s| s.moment), two files sharing a timestamp could be ordered arbitrarily between runs. Both paths now sort by the timestamp-prefixed filename. This is a small deliberate behaviour change, in the direction of determinism.
• GQL literals needed escaping. The tag pattern admits apostrophes, so @don't terminated the literal early.

Follow-up for the user

The stream folder is a git repo with no ignore entry for the cache; .streams-cache.grafeo* should be added there (it is already ignored in this repo). It should also be excluded from OneDrive sync — it is a binary rewritten on every note change, so syncing it means constant upload churn and cross-machine conflict copies. No cache file was left behind on the notes folder.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

Implements #142. Caches parsed stream files in an embedded [Grafeo](https://grafeo.dev/) graph at `.streams-cache.grafeo` inside the stream folder. Commands validate it with `stat` alone and re-read only files that actually changed. ## Design The graph is split into two layers, which is what makes the change safe: - **Structural** (`:File`, `:Shard`, `HAS_ROOT`/`HAS_CHILD`/`IN_FILE`) — stores raw `Shard` trees. Independent of any `RepositoryConfiguration` or timezone. - **Projection** (`:Tag`, `:DimensionValue`, `:Day`, `TAGGED`/`PLACED_IN`/`ON_DAY`) — derived by localizing under the canonical merged configuration. Purely additive; only graph queries read it. Commands read the *structural* layer and re-run the pure `localize_stream_file` with **their own** configuration and timezone. That matters because the same folder is loaded three different ways today (`todo`/`edit` use `TaskConfiguration` at UTC, `timesheet` uses `BasicTimesheetConfiguration` at the repo timezone, the LSP merges both). Caching `LocalizedShard`s directly would have forced config and tz into the cache key; this way output is unchanged. Everything is derived state: a corrupt cache, an unknown schema version, or a lock held by another process all resolve to rebuilding or falling back. **A cache failure can never fail a command.** ## Verified On the real 126-file corpus, `todo` and `timesheet` are **byte-identical** with and without the cache. All four build targets pass: | Target | Result | |---|---| | `nix flake check` | passed | | `streamd-musl` | `static-pie linked, stripped` | | `streamd-windows` | `PE32+ executable` | | `streamd-deb` | package created | 261 tests pass (up from 211 on `main` — 50 new), including equivalence across both configurations × two timezones on cold and warm caches, corruption recovery, schema bump, deletion, and lock contention. ## Performance: no local win Measured on this machine (126 files, 235 KB, local SSD): | | time | |---|---| | uncached | 13 ms | | cold (builds cache) | 25 ms | | warm | 14 ms | **The warm path is 1 ms slower than not caching at all.** Parsing 235 KB is already trivial, so graph open + stat + reconstruct costs slightly more than it saves. This is consistent with the issue's premise — the target is the WSL/OneDrive share where per-file reads dominate — but it means the benefit is still unproven. The honest success criterion remains a before/after warm run on the work machine; if it does not pay off there, `STREAMD_NO_CACHE=1` and `streamd cache clear` make it a no-op. ## Deviations from the plan Three, each with a reason: 1. **Feature set is `["edge", "storage"]`, not `["gql"]`.** The planned line enables neither the LPG property-graph store nor persistence — `GrafeoDB::open` does not exist under it. Dependency tree is 46 crates with no `*-sys`/`cc`/`bindgen`; neither allocator feature is enabled. 2. **No incremental subgraph update on `didSave`/`didChangeWatchedFiles`.** Redundant: invalidation is stat-based, so the next process to open the cache re-reads just that file. The graph is self-correcting. 3. **`:Day` nodes added.** The plan listed `files_touching_date`, but dates are localization *output*, not structural facts. Modelling days as nodes keeps canonical moments out of the config-independent layer while still making date questions a traversal. ## Notable fixes found while building - **Lock contention destroyed the cache.** `open_or_rebuild` treated *any* open failure as corruption and deleted the file — so a second process would wipe a database the first was actively using. Lock contention is now distinguished and propagated untouched. Pinned by `test_a_locked_cache_is_not_deleted`. - **`load_markdown_shards` was order-nondeterministic.** It returned `WalkDir` order; since consumers use a *stable* `sort_by_key(|s| s.moment)`, two files sharing a timestamp could be ordered arbitrarily between runs. Both paths now sort by the timestamp-prefixed filename. This is a small deliberate behaviour change, in the direction of determinism. - **GQL literals needed escaping.** The tag pattern admits apostrophes, so `@don't` terminated the literal early. ## Follow-up for the user The stream folder is a git repo with no ignore entry for the cache; `.streams-cache.grafeo*` should be added there (it is already ignored in *this* repo). It should also be excluded from OneDrive sync — it is a binary rewritten on every note change, so syncing it means constant upload churn and cross-machine conflict copies. No cache file was left behind on the notes folder. https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel added 10 commits 2026-09-06 18:14:29 +02:00

feat(cache): add grafeo dependency and cache file lifecycle ... 26eb91aa2a

Introduces the Grafeo embedded graph database and the `store` module that
owns the cache location.

The feature set is `["edge", "storage"]`, not the `["gql"]` named in the
plan: `gql` alone maps only to `grafeo-engine/gql` and enables neither the
LPG property-graph store nor persistence, so `GrafeoDB::open` does not
exist under it. `edge` supplies the LPG store plus GQL, `storage` supplies
wal + grafeo-file. Neither allocator feature is enabled, keeping the
dependency tree free of C code (46 crates, no `*-sys`/`cc`/`bindgen`).

`remove_cache` deletes either a file or a directory at the cache path.
Verified that `.grafeo` currently yields a single file, but the format is
extension-driven in a pre-1.0 crate, so the directory case is handled too.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cache): persist and reconstruct shard trees in the graph ... 2887ce9104

Adds the structural layer: `:File` and `:Shard` nodes joined by HAS_ROOT,
HAS_CHILD and IN_FILE edges, plus the schema constants, a stable FNV-1a
projection fingerprint, and Value conversion helpers.

This layer is deliberately config- and timezone-independent. It stores the
raw parsed `Shard` tree, so a reader can re-run the pure `localize_stream_file`
under whichever configuration and timezone the calling command uses. That is
what keeps command output byte-identical while still skipping the file reads.

Sibling order is preserved with an explicit `ordinal` property rather than
relying on edge enumeration order. Files that fail to parse are stored with
`parse_ok = false` so they are not re-read on every run.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cache): stat-based incremental sync against the filesystem ... b9b854edfa

`sync_structural` diffs the graph against a stat-only scan: files whose
(size, mtime_ns) still match are left untouched and never opened, changed
files are re-read, and files that vanished have their subgraphs deleted.

`SyncStats::read_from_disk()` makes the guarantee testable rather than
assumed -- `test_second_sync_reads_nothing_when_nothing_changed` asserts it
is exactly zero for an unchanged corpus, which is the entire point of the
feature on the OneDrive-backed share.

The scan is recursive and skips dot directories, matching the LSP's
`list_markdown_files`. `filetime` is added as a dev-dependency so the
"only the changed file is re-read" test does not depend on wall-clock
timing or filesystem mtime granularity.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cache): public API with corruption, schema and lock recovery ... fa2bc72c4e

Adds `load_stream_files`, `rebuild`, `clear` and `status`, plus the
`STREAMD_NO_CACHE` switch.

Fixes a data-loss bug caught by the lock-contention test: `open_or_rebuild`
treated *any* failure to open as corruption and deleted the cache, so a
second process would wipe the database a first one was actively using.
Lock contention is now distinguished from corruption and propagated to the
caller untouched, leaving the file intact for its owner.

Opening a freshly created cache now stamps the `:Meta` node in place rather
than deleting and recreating the file it just made.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cli): load the corpus through the cache in todo, edit, daily, timesheet ... ddd3d77906

`load_markdown_shards_cached` syncs the graph, then localizes cached shard
trees with the caller's own configuration and timezone, so results are
identical to the uncached loader while skipping the file reads. Any cache
failure -- lock contention, corruption, an unwritable folder -- falls back
to `load_markdown_shards` silently.

Also makes `load_markdown_shards` sort by file name. It previously returned
whatever order WalkDir produced, which is filesystem-dependent; since
consumers use a *stable* `sort_by_key(|s| s.moment)`, two files sharing a
timestamp could be ordered arbitrarily and differ between runs. Sorting on
the timestamp-prefixed names makes both paths deterministic and equal.

The equivalence is pinned by tests across both preconfigured configurations
and two timezones, on cold and warm caches, after an edit, and under lock
contention.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cli): add streamd cache status, rebuild and clear ... f7965ce7ad

Gives the cache a user-facing handle: `status` reports its location, size,
schema version and file count, `rebuild` discards and reindexes, `clear`
deletes it. `status` is the default when no subcommand is given.

`status` degrades rather than failing when the cache is locked by another
process or unreadable, since its job is to explain the situation.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(cache): projection layer with tag, dimension and day nodes ... 7dca6f9acd

Adds the second graph layer: `:Tag`, `:DimensionValue` and `:Day` nodes,
joined to shards by TAGGED, PLACED_IN and ON_DAY, built by localizing cached
trees under the canonical merged configuration.

This is what makes the store a graph rather than a serialized tree. It is
purely additive -- the commands read the structural layer and localize with
their own settings -- so it cannot change what `todo` or `timesheet` print.

Four queries land with it: `shards_in_dimension` (one hop, where
`find_shard_by_position` walks every node of every file),
`tags_co_occurring_with`, `dimension_value_counts`, and `files_touching_date`,
which answers the cross-file question the LSP currently recomputes from
scratch on every diagnostic request.

`:Day` nodes exist because dates are localization output, not structural
facts; modelling them as nodes keeps canonical moments out of the
config-independent layer while still making date questions a traversal.

Values interpolated into GQL are escaped: the tag pattern admits apostrophes,
so `@don't` would otherwise terminate the literal early.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

style(cache): move value helpers above the test module ... a5c2989723

Clippy's `items_after_test_module` fires under --all-targets, which
`nix flake check` runs.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

feat(lsp): seed the file cache from the graph at startup ... ca7fa556f8

`LspState::new` now pre-populates `file_cache` from the local graph cache
instead of leaving every entry to be filled lazily by reading and parsing
each file. That first workspace-wide request is the cold-start cost that
hurts on a network-backed share.

Localization still runs with the LSP's own merged configuration, so seeded
entries are identical to parsing from disk -- a test asserts that directly.
Seeding is best-effort: when the cache is locked or unusable the caches stay
empty and every existing lazy path behaves exactly as before.

No incremental subgraph update is wired into `did_change_watched_files`.
It would be redundant: the sync is stat-based, so the next process to open
the cache detects the change and re-reads just that file. The graph is
self-correcting rather than something the LSP must keep in step.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

docs: document the local graph cache in README and REQUIREMENTS ... d17d98e0fa

Adds a "Local Cache" section to the README covering the `streamd cache`
commands, `STREAMD_NO_CACHE`, and why the cache should be excluded from
both version control and file sync -- a binary file rewritten on every note
change produces upload churn and cross-machine conflict copies.

Adds R26 to REQUIREMENTS covering the file location and why the `.grafeo`
extension is load-bearing, the two-layer model, the stat-only invalidation
contract, the exclusive-lock and fallback rules, the `cache` subcommand, and
the LSP's cold-start seeding.

States plainly that the cache buys little on local disk and is aimed at
network-backed shares, so the tradeoff is not overstated.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel referenced this pull request 2026-09-06 18:16:21 +02:00

feat(cache): local graph-database cache for localized shards (Grafeo) #142

kfickel added 1 commit 2026-09-06 19:18:47 +02:00

refactor(cache): store caches in XDG cache dir, one per stream folder ... 603d240f52

Moves the cache out of the stream folder and into `~/.cache/streamd`
(`ProjectDirs::cache_dir()`). Notes directories are version-controlled and
file-synced; a binary rewritten on every edit caused sync churn and
cross-machine conflict copies there, and needed a .gitignore entry in every
stream repo. Nothing is written beside the markdown now.

The cache is XDG *cache*, not *config*: it is rebuildable derived state that
should be safe to delete and pointless to back up.

Each stream folder gets its own database, named `<slug>-<hash>.grafeo` where
the hash is FNV-1a over the folder's canonical absolute path. Multiple
streamd instances with different config files therefore never share a cache,
and folders sharing a basename do not collide. Canonicalizing first means
`.`, `..`, a trailing slash and symlinks all resolve to one cache. The slug
is for humans browsing the cache dir and never affects identity.

Drops the `spill` feature (`storage` = wal + grafeo-file + spill + mmap
becomes wal + grafeo-file). Spill created a `.grafeo.spill` sidecar
directory next to every database, so the cache was not the single file it
was supposed to be; without it there is exactly one file per stream folder.
`remove_cache` still sweeps `.spill`/`.wal` sidecars so caches written by a
differently-configured build leave nothing orphaned.

Adds `STREAMD_CACHE_DIR` to relocate the cache directory, which also keeps
tests out of the developer's real cache.

Claude-Session: https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel commented 2026-09-06 19:19:04 +02:00

Author

Owner

Copy link

Update: cache relocated to the XDG cache directory

Pushed 603d240. The cache no longer lives in the stream folder.

Location is now ~/.cache/streamd/<slug>-<hash>.grafeo via ProjectDirs::cache_dir(). XDG cache, not config — it is rebuildable derived state that should be safe to delete and pointless to back up.

One database per stream folder. The name is <slug>-<hash>.grafeo, where the hash is FNV-1a over the folder's canonical absolute path. Multiple streamd instances with different config files therefore never share a cache, and two folders sharing a basename do not collide. Canonicalizing first means ., .., a trailing slash and symlinks all resolve to one cache rather than four. The slug is for humans browsing the cache directory and never affects identity.

STREAMD_CACHE_DIR overrides the directory; it also keeps tests out of the developer's real cache.

The relocation surfaced a bug in the original design

Grafeo's spill feature creates a .grafeo.spill sidecar directory beside every database. The cache was therefore never the single file it was documented to be, and remove_cache was orphaning the sidecar on every clear — including inside the stream folder, where it left an empty directory behind after the earlier version ran.

Fixed by dropping spill from the feature set (storage = wal + grafeo-file + spill + mmap → wal + grafeo-file). There is now exactly one file per stream folder, verified on the real corpus. remove_cache still sweeps .spill/.wal sidecars defensively, so a cache written by a differently-configured build leaves nothing orphaned. I also removed the stray .spill directory the earlier build left in the notes folder.

Re-verified

• todo and timesheet byte-identical with and without the cache, on the real 126-file corpus
• ~/.cache/streamd/ holds exactly one file; the stream folder is untouched
• 267 tests pass (up from 266 — added a sidecar-sweep test)
• nix flake check, streamd-musl, streamd-windows, streamd-deb all pass

Docs

README's "Local Cache" section rewritten for the new location, and the .gitignore / sync-exclusion advice removed — there is nothing to exclude any more. R26 in REQUIREMENTS updated with the path scheme, canonicalization and per-folder separation rules. The repo's own .gitignore entry is gone.

The earlier "Follow-up for the user" note in the PR description is obsolete — no .gitignore entry or OneDrive exclusion is needed now.

https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

## Update: cache relocated to the XDG cache directory Pushed `603d240`. The cache no longer lives in the stream folder. **Location** is now `~/.cache/streamd/<slug>-<hash>.grafeo` via `ProjectDirs::cache_dir()`. XDG *cache*, not *config* — it is rebuildable derived state that should be safe to delete and pointless to back up. **One database per stream folder.** The name is `<slug>-<hash>.grafeo`, where the hash is FNV-1a over the folder's **canonical** absolute path. Multiple streamd instances with different config files therefore never share a cache, and two folders sharing a basename do not collide. Canonicalizing first means `.`, `..`, a trailing slash and symlinks all resolve to one cache rather than four. The slug is for humans browsing the cache directory and never affects identity. `STREAMD_CACHE_DIR` overrides the directory; it also keeps tests out of the developer's real cache. ### The relocation surfaced a bug in the original design Grafeo's `spill` feature creates a **`.grafeo.spill` sidecar directory** beside every database. The cache was therefore never the single file it was documented to be, and `remove_cache` was orphaning the sidecar on every `clear` — including inside the stream folder, where it left an empty directory behind after the earlier version ran. Fixed by dropping `spill` from the feature set (`storage` = wal + grafeo-file + spill + mmap → `wal` + `grafeo-file`). There is now exactly one file per stream folder, verified on the real corpus. `remove_cache` still sweeps `.spill`/`.wal` sidecars defensively, so a cache written by a differently-configured build leaves nothing orphaned. I also removed the stray `.spill` directory the earlier build left in the notes folder. ### Re-verified - `todo` and `timesheet` byte-identical with and without the cache, on the real 126-file corpus - `~/.cache/streamd/` holds exactly one file; the stream folder is untouched - 267 tests pass (up from 266 — added a sidecar-sweep test) - `nix flake check`, `streamd-musl`, `streamd-windows`, `streamd-deb` all pass ### Docs README's "Local Cache" section rewritten for the new location, and the `.gitignore` / sync-exclusion advice removed — there is nothing to exclude any more. R26 in REQUIREMENTS updated with the path scheme, canonicalization and per-folder separation rules. The repo's own `.gitignore` entry is gone. **The earlier "Follow-up for the user" note in the PR description is obsolete** — no `.gitignore` entry or OneDrive exclusion is needed now. https://claude.ai/code/session_012figbDQRDBcAx11FYarFBu

kfickel merged commit 603d240f52 into main 2026-09-06 21:08:09 +02:00

kfickel deleted branch 142_graph-database-cache 2026-09-06 21:08:09 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

planned

The feature is ready to be implemented by an AI agent

No labels

planned

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

kfickel/streamd!143

No description provided.