feat: list and complete used tags and markers (v0.7.0) #150

Merged

kfickel merged 10 commits from 149_used-tags-markers-completion into main 2026-09-14 13:55:38 +02:00

Conversation 1 Commits 10 Files changed 19 +869 -60

kfickel commented 2026-09-14 13:42:54 +02:00

Owner

Copy link

Closes #149.

What this adds

streamd tags

• Prints every tag and marker used anywhere in the stream as @name, one per line, sorted alphabetically.
• Subfolders are included, and a marker counts whether or not it is configured.
• Names made only of digits (@20260101, @0900) are left out.
• The list is read from the graph cache. If the cache is locked or STREAMD_NO_CACHE is set, the corpus is scanned directly instead, and a test checks that both paths print the same list.

LSP @ completion

• Used names are suggested after configured markers and before the date/time snippets.
• The prefix match ignores case, and names that are already configured markers are not repeated.
• The list lives in memory, so a completion request never waits on the disk.
• It is loaded on start and on .streamd.toml reload, then refreshed in the background after every save and every watched .md change.
• If the cache can't be used, the list is read from the workspace files, reusing buffers the server has already parsed.

Graph cache

• Markers are now projected too. They share the :Tag node of the same name but use a separate MARKED edge, so tag co-occurrence keeps its meaning.
• A projection_dirty flag on :Meta records that files changed since the projection was built. Every command syncs the structural layer, but only these new consumers rebuild the projection, so a single sync result can't tell whether the projection is stale.
• The new consumers rebuild a stale projection on demand, so streamd cache rebuild is no longer needed first. Other commands never touch the projection.
• There is no schema bump. A projection built before this change has no dirty flag, so it counts as stale and is rebuilt.

Also fixed: flaky tests (the failing Nix / home-manager build)

• Two tests changed STREAMD_CACHE_DIR and STREAMD_NO_CACHE process-wide while the rest of the suite ran in parallel. A cache test could briefly see another cache directory or a disabled cache.
• That broke the v0.6.0 Nix build in test_cache_lives_outside_the_stream_folder, and occasionally a new LSP test.
• The logic is now tested by passing the value in (disables_cache, load_markdown_shards_with_cache), so no test changes the environment anymore.
• The full suite passed 20 runs in a row afterwards.

Version

Bumped to 0.7.0 (minor).

Verification

• cargo test: 313 tests pass. cargo clippy --all-targets and cargo fmt are clean, and nix flake check passes.
• Manual streamd tags run: output is the same with a cold cache, a warm cache and STREAMD_NO_CACHE=1.
• The LSP was driven over stdio against a throwaway workspace:
  • @ru suggested @rustacean, a tag used only in another file.
  • After saving a draft containing @brandnew, @bra suggested it.
  • The server exits cleanly once stdin closes.

Follow-ups (not in this PR)

• STREAMD_BASE_FOLDER (R23) is not implemented. Settings::load never reads it; only config.toml and the current directory are used.
• Tags keep trailing punctuation. The tag pattern (R1) matches @Idea:, @TNG? and @Name,, so the list and completion show near-duplicates. Fixing that belongs in extraction, not in this listing.

https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

Closes #149. ## What this adds **`streamd tags`** - Prints every tag and marker used anywhere in the stream as `@name`, one per line, sorted alphabetically. - Subfolders are included, and a marker counts whether or not it is configured. - Names made only of digits (`@20260101`, `@0900`) are left out. - The list is read from the graph cache. If the cache is locked or `STREAMD_NO_CACHE` is set, the corpus is scanned directly instead, and a test checks that both paths print the same list. **LSP `@` completion** - Used names are suggested after configured markers and before the date/time snippets. - The prefix match ignores case, and names that are already configured markers are not repeated. - The list lives in memory, so a completion request never waits on the disk. - It is loaded on start and on `.streamd.toml` reload, then refreshed in the background after every save and every watched `.md` change. - If the cache can't be used, the list is read from the workspace files, reusing buffers the server has already parsed. **Graph cache** - Markers are now projected too. They share the `:Tag` node of the same name but use a separate `MARKED` edge, so tag co-occurrence keeps its meaning. - A `projection_dirty` flag on `:Meta` records that files changed since the projection was built. Every command syncs the structural layer, but only these new consumers rebuild the projection, so a single sync result can't tell whether the projection is stale. - The new consumers rebuild a stale projection on demand, so `streamd cache rebuild` is no longer needed first. Other commands never touch the projection. - There is no schema bump. A projection built before this change has no dirty flag, so it counts as stale and is rebuilt. ## Also fixed: flaky tests (the failing Nix / home-manager build) - Two tests changed `STREAMD_CACHE_DIR` and `STREAMD_NO_CACHE` process-wide while the rest of the suite ran in parallel. A cache test could briefly see another cache directory or a disabled cache. - That broke the v0.6.0 Nix build in `test_cache_lives_outside_the_stream_folder`, and occasionally a new LSP test. - The logic is now tested by passing the value in (`disables_cache`, `load_markdown_shards_with_cache`), so no test changes the environment anymore. - The full suite passed 20 runs in a row afterwards. ## Version Bumped to **0.7.0** (minor). ## Verification - `cargo test`: 313 tests pass. `cargo clippy --all-targets` and `cargo fmt` are clean, and `nix flake check` passes. - Manual `streamd tags` run: output is the same with a cold cache, a warm cache and `STREAMD_NO_CACHE=1`. - The LSP was driven over stdio against a throwaway workspace: - `@ru` suggested `@rustacean`, a tag used only in another file. - After saving a draft containing `@brandnew`, `@bra` suggested it. - The server exits cleanly once stdin closes. ## Follow-ups (not in this PR) - **`STREAMD_BASE_FOLDER` (R23) is not implemented.** `Settings::load` never reads it; only `config.toml` and the current directory are used. - **Tags keep trailing punctuation.** The tag pattern (R1) matches `@Idea:`, `@TNG?` and `@Name,`, so the list and completion show near-duplicates. Fixing that belongs in extraction, not in this listing. https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

kfickel added 8 commits 2026-09-14 13:42:54 +02:00

feat(cache): project marker names and query every used tag and marker a0e1e4316c

feat(cache): rebuild the projection on demand when files changed ... 31133164c0

Every command syncs the structural layer but only some rebuild the
projection, so a dirty flag on the cache records the drift across runs.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

feat(cli): add streamd tags listing every used tag and marker ... 8be86e2c1b

Reads the graph cache and falls back to scanning the corpus when the
cache is locked or disabled, so both paths print the same list.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

feat(lsp): complete tags and markers used anywhere in the workspace ... 8ff59d2fab

The list lives in memory and is refreshed from the graph cache on start,
config reload, save and watched file changes, so completion never waits.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

docs: describe streamd tags, used-name completion and projection freshness ... 8068b2e716

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

fix(test): stop tests from mutating process-wide environment variables ... ddf45e81f3

Two tests set STREAMD_CACHE_DIR and STREAMD_NO_CACHE with set_var while
the rest of the suite ran in parallel, so any cache test could briefly see
another directory or a disabled cache. This failed the Nix build in
test_cache_lives_outside_the_stream_folder. The logic is now tested by
passing the value in instead.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

fix(lsp): read used names from the workspace when the cache is unusable ... c9d229468c

The fallback only saw files the server had already parsed, so a locked or
disabled cache left completion without most of the workspace's names.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

bump: v0.7.0 ... 6fb2cfba60

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

kfickel referenced this pull request 2026-09-14 13:43:09 +02:00

Include used Tags / Markers in LSP Auto-Completion #149

kfickel added 2 commits 2026-09-14 13:48:54 +02:00

fix(config): honour STREAMD_BASE_FOLDER as documented in R23 ... 34828aa64f

Settings::load only ever read config.toml, so the override silently did
nothing and commands ran against the configured stream folder instead. A
non-empty value now wins and the config file is not read; an empty value is
ignored. The value is passed in so tests never set the environment.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

fix(extract): strip trailing punctuation from tag and marker names ... 6db54ed67a

Prose like "Call @Anna, then @Bob." produced the names "Anna," and "Bob.",
so `streamd tags` and completion listed near-duplicates such as @Idea and
@Idea:. Inner punctuation (@sim.de, @don't) is kept, and the stripped
suffix does not end a marker run. The cache schema is bumped so unchanged
files are re-parsed instead of keeping their old names.

Claude-Session: https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

kfickel commented 2026-09-14 13:51:05 +02:00

Author

Owner

Copy link

Follow-ups now fixed in this PR

Both items listed under Follow-ups in the description are fixed here, each in its own commit.

34828aa fix(config): honour STREAMD_BASE_FOLDER as documented in R23

• Settings::load never read the variable, so the override did nothing and commands ran against the folder from config.toml.
• A non-empty value now takes precedence, and config.toml is not read at all then. An empty value is ignored.
• The value is passed into Settings::load_from, so the tests don't set environment variables (avoiding the kind of flakiness fixed earlier in this PR).
• Checked with the real binary in throwaway folders: the override is used, and an empty value falls back.

6db54ed fix(extract): strip trailing punctuation from tag and marker names

• Call @Anna, then @Bob. now yields Anna and Bob, and (@Klein) yields Klein. Previously streamd tags and completion showed near-duplicates like @Idea / @Idea:.
• Stripped: any run of . , : ; ! ? ) } " ' at the end of the name. Inner punctuation is kept (@sim.de, @Until:20250731, @don't).
• The stripped characters don't count as content, so @Task, @Done fix still has two markers.
• An @ followed only by punctuation (@?!) is plain text, not a tag.
• The cache schema is bumped to 2, so every file is re-parsed once after upgrading. Otherwise unchanged files would keep their cached names with punctuation.

Both are documented in REQUIREMENTS.md (R1, R23), and the override is also in README.md. cargo test passes (326 tests, 5 runs in a row), clippy is clean, and nix flake check passes.

https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

## Follow-ups now fixed in this PR Both items listed under **Follow-ups** in the description are fixed here, each in its own commit. **`34828aa` fix(config): honour `STREAMD_BASE_FOLDER` as documented in R23** - `Settings::load` never read the variable, so the override did nothing and commands ran against the folder from `config.toml`. - A non-empty value now takes precedence, and `config.toml` is not read at all then. An empty value is ignored. - The value is passed into `Settings::load_from`, so the tests don't set environment variables (avoiding the kind of flakiness fixed earlier in this PR). - Checked with the real binary in throwaway folders: the override is used, and an empty value falls back. **`6db54ed` fix(extract): strip trailing punctuation from tag and marker names** - `Call @Anna, then @Bob.` now yields `Anna` and `Bob`, and `(@Klein)` yields `Klein`. Previously `streamd tags` and completion showed near-duplicates like `@Idea` / `@Idea:`. - Stripped: any run of `. , : ; ! ? ) } " '` at the end of the name. Inner punctuation is kept (`@sim.de`, `@Until:20250731`, `@don't`). - The stripped characters don't count as content, so `@Task, @Done fix` still has two markers. - An `@` followed only by punctuation (`@?!`) is plain text, not a tag. - The cache schema is bumped to 2, so every file is re-parsed once after upgrading. Otherwise unchanged files would keep their cached names with punctuation. Both are documented in `REQUIREMENTS.md` (R1, R23), and the override is also in `README.md`. `cargo test` passes (326 tests, 5 runs in a row), clippy is clean, and `nix flake check` passes. https://claude.ai/code/session_013fFsc2ryYqZRBUorNdbWZw

kfickel merged commit 6db54ed67a into main 2026-09-14 13:55:38 +02:00

kfickel deleted branch 149_used-tags-markers-completion 2026-09-14 13:55:39 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

planned

The feature is ready to be implemented by an AI agent

No labels

planned

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

kfickel/streamd!150

No description provided.