Include used Tags / Markers in LSP Auto-Completion #149

New issue

Closed

opened 2026-09-14 13:15:24 +02:00 by kfickel · 2 comments

kfickel commented 2026-09-14 13:15:24 +02:00

Owner

Copy link

• Command to show all used tags/markers (together), sorted alphabetically
• Supply Markers + tags in LSP-autocompletion

Use the graph-cache, it needs to be fast!

* Command to show all used tags/markers (together), sorted alphabetically * Supply Markers + tags in LSP-autocompletion Use the graph-cache, it needs to be fast!

kfickel commented 2026-09-14 13:22:56 +02:00

Author

Owner

Copy link

Implementation Plan

Clarified requirements

• "Used" means actually written. Every @word that shows up in the corpus counts, as a marker (before content) or as a tag (after content), whether or not it is declared in RepositoryConfiguration.
• Recursive scope. Files in subdirectories count too, matching the cache's structural layer and the LSP. This differs on purpose from the top-level-only view of todo/timesheet.
• The projection rebuilds itself when stale. This happens only on the code paths added here. Existing commands (todo, timesheet, …) keep their current cost.
• LSP refresh. The used-words set is loaded on server start and on .streamd.toml reload, and refreshed on every markdown save or watched .md change.
• Temporal markers are excluded. All-digit names (@20260101, @0900, R16) are left out of both the list and completion. The existing date/time snippets already cover them.

Current state (for reference)

• src/cache/projection.rs::project_shard only interns localized.tags as :Tag nodes (TAGGED). Raw marker names never reach the graph. Markers only show up indirectly, as :DimensionValue placements, and only when they are configured.
• The projection is built only by cache::rebuild() (streamd cache rebuild). cache::load_stream_files syncs the structural layer but never updates the projection, so the projection is usually stale or missing.
• src/cli/commands/lsp.rs::completions_for_line only suggests config.markers (R25a).
• The README says graph queries have "no CLI surface … yet".

1. Graph schema / projection (src/cache/)

1. schema.rs

   • Add EDGE_MARKED: &str = "MARKED".
   • Add PROP_PROJECTION_DIRTY: &str = "projection_dirty".
   • Bump SCHEMA_VERSION to 2, so existing caches are discarded and rebuilt in the new shape.

2. projection.rs::project_shard: also loop over localized.markers, intern each one with the existing intern_tag and create an EDGE_MARKED edge from the shard. A name used both ways therefore shares a single :Tag node. tags_co_occurring_with only follows TAGGED, so it keeps its current meaning (add a test for that).

3. Staleness tracking: sync_structural also runs when other commands load the corpus, so its own SyncStats can't tell whether the projection is stale.

   • sync.rs::sync_structural: when added + updated + removed > 0, set projection_dirty = true on :Meta.
   • projection.rs::build_projection: after stamp_fingerprint, set projection_dirty = false.
   • Add a helper projection_is_fresh(session, config, tz) -> bool. It returns true only if the fingerprint matches and projection_dirty is false (a missing property counts as dirty).

4. query.rs: add used_tag_and_marker_names(session) -> Result<Vec<String>, StreamdError>. It runs MATCH (t:Tag) RETURN t.name, then dedups, drops all-digit names, and sorts alphabetically. Sort with plain str ordering first, for deterministic output.

5. mod.rs: add pub fn load_used_names(base_folder: &Path, tz: Tz) -> Result<Vec<String>, StreamdError>. Steps:

   1. store::open_or_rebuild
   2. sync::sync_structural
   3. build_projection(canonical_configuration(), tz) if !projection_is_fresh
   4. query::used_tag_and_marker_names

   It returns Err when the cache is locked or unusable, and callers fall back.

2. CLI command streamd tags

• src/cli/args.rs: add Commands::Tags with the doc comment "List all tags and markers used in the stream, alphabetically".
• src/cli/commands/tags.rs (new; register it in commands/mod.rs and dispatch it in main.rs/cli/mod.rs the same way Cache is):
  • run() loads Settings, then calls run_in(base_folder). run_in takes the timezone from load_repository_config(base_folder)?.timezone_or_utc(), same as cache rebuild.
  • collect_used_names(base_folder, tz) -> Result<Vec<String>, StreamdError>:
    • Use cache::load_used_names unless cache::is_disabled().
    • On Err, or when the cache is disabled, fall back to a direct recursive scan. Use cache::sync::scan_disk for the paths, then parse_markdown_file, and collect markers + tags from the parsed Shard tree (no localization needed). Apply the same all-digit filter, dedup and sort.
    • Put the filter/dedup/sort in one shared helper (e.g. normalize_used_names(impl Iterator<Item=String>)) so both paths produce identical output.
  • Output: one name per line, prefixed with @, to stdout. Print nothing for an empty corpus.

3. LSP completion (src/cli/commands/lsp.rs)

1. LspState: add used_names: std::sync::RwLock<Vec<String>>.

2. LspState::refresh_used_names(&self):

   • Try cache::load_used_names(&self.base_folder, self.tz), unless the cache is disabled.
   • On failure, fall back to collecting markers + tags from every entry currently in file_cache (walk the LocalizedShard trees), normalized with the same helper. Either way no disk reads happen on this path, and it never fails.
   • Store the result in used_names.

3. Call sites:

   • build_state, after seed_from_cache() (covers start-up and .streamd.toml reload).
   • did_save.
   • The markdown branch of did_change_watched_files.

   Run the refresh via tokio::task::spawn_blocking (clone the Arc<LspState>) so the graph rebuild never blocks request handling. A completion request that arrives meanwhile just sees the previous set.

4. completions_for_line: add a parameter used_names: &[String], keeping the function pure and unit-testable. After the config markers, add an item for each used name that:

   • matches the typed prefix, case-insensitively, like today;
   • is not already in config.markers (no duplicate labels).

   Each item gets label: "@name", insert_text: name, kind: CompletionItemKind::TEXT, detail: "Used in stream" and sort_text: "1b_{name}". Resulting order: conditional config markers (0_), then config markers (1_), then used names (1b_), then date/time snippets (2_).

5. The completion handler passes &state.used_names.read().

4. Tests

• cache/projection.rs / cache/query.rs:
  • Markers, including unconfigured ones like @Project-X, appear in used_tag_and_marker_names.
  • A name used as both marker and tag is listed once.
  • All-digit names are excluded and output is sorted.
  • tags_co_occurring_with is unaffected by MARKED edges.
• Staleness:
  • Build the projection, then run sync_structural after editing or adding a file: projection_is_fresh must be false.
  • After load_used_names, the new tag is returned.
  • An unchanged corpus does not rebuild the projection. Check that the dirty flag stays false and the fingerprint is unchanged.
• cli/commands/tags.rs:
  • Cached and uncached (STREAMD_NO_CACHE=1 / locked cache) paths return identical results on a corpus with nested files, unconfigured markers, content tags and a temporal marker.
  • Nested files are included.
• lsp.rs:
  • completions_for_line suggests a used tag matching the prefix.
  • It does not duplicate a used name that is also a config marker.
  • Used names sort after config markers and before the date/time snippets.
  • Existing tests keep passing, updated for the new parameter (pass &[]).
• cache/mod.rs: load_used_names on a fresh cache builds the projection and returns names.

5. Docs

• REQUIREMENTS.md:
  • Add streamd tags to R20.
  • Add a short R-section for it, covering scope (recursive, markers + tags, digits excluded, alphabetical, @-prefixed, cache with direct-scan fallback).
  • Extend R25a with used-name completions (source, dedup against config, sort order, refresh cadence).
  • Update R26a: MARKED edge, projection_dirty, auto-rebuild when stale for these consumers, schema v2.
• README.md:
  • Add streamd tags to the Commands list.
  • Update the LSP feature table row for @ completions to mention used tags/markers.
  • Change the "no CLI surface is exposed for them yet" sentence to mention streamd tags.

6. Verification

• cargo test, cargo clippy, cargo fmt, then nix flake check.
• Manual check: run streamd tags on a sample stream. Start streamd lsp in an editor, type @ followed by a prefix of a tag used only in another file, and confirm it is suggested. Add a new tag, save, and confirm it becomes suggestible.

## Implementation Plan ### Clarified requirements - **"Used" means actually written.** Every `@word` that shows up in the corpus counts, as a marker (before content) or as a tag (after content), whether or not it is declared in `RepositoryConfiguration`. - **Recursive scope.** Files in subdirectories count too, matching the cache's structural layer and the LSP. This differs on purpose from the top-level-only view of `todo`/`timesheet`. - **The projection rebuilds itself when stale.** This happens only on the code paths added here. Existing commands (`todo`, `timesheet`, …) keep their current cost. - **LSP refresh.** The used-words set is loaded on server start and on `.streamd.toml` reload, and refreshed on every markdown save or watched `.md` change. - **Temporal markers are excluded.** All-digit names (`@20260101`, `@0900`, R16) are left out of both the list and completion. The existing date/time snippets already cover them. ### Current state (for reference) - `src/cache/projection.rs::project_shard` only interns `localized.tags` as `:Tag` nodes (`TAGGED`). Raw marker names never reach the graph. Markers only show up indirectly, as `:DimensionValue` placements, and only when they are configured. - The projection is built only by `cache::rebuild()` (`streamd cache rebuild`). `cache::load_stream_files` syncs the structural layer but never updates the projection, so the projection is usually stale or missing. - `src/cli/commands/lsp.rs::completions_for_line` only suggests `config.markers` (R25a). - The README says graph queries have "no CLI surface … yet". ### 1. Graph schema / projection (`src/cache/`) 1. `schema.rs` - Add `EDGE_MARKED: &str = "MARKED"`. - Add `PROP_PROJECTION_DIRTY: &str = "projection_dirty"`. - Bump `SCHEMA_VERSION` to `2`, so existing caches are discarded and rebuilt in the new shape. 2. `projection.rs::project_shard`: also loop over `localized.markers`, intern each one with the existing `intern_tag` and create an `EDGE_MARKED` edge from the shard. A name used both ways therefore shares a single `:Tag` node. `tags_co_occurring_with` only follows `TAGGED`, so it keeps its current meaning (add a test for that). 3. Staleness tracking: `sync_structural` also runs when *other* commands load the corpus, so its own `SyncStats` can't tell whether the projection is stale. - `sync.rs::sync_structural`: when `added + updated + removed > 0`, set `projection_dirty = true` on `:Meta`. - `projection.rs::build_projection`: after `stamp_fingerprint`, set `projection_dirty = false`. - Add a helper `projection_is_fresh(session, config, tz) -> bool`. It returns true only if the fingerprint matches **and** `projection_dirty` is `false` (a missing property counts as dirty). 4. `query.rs`: add `used_tag_and_marker_names(session) -> Result<Vec<String>, StreamdError>`. It runs `MATCH (t:Tag) RETURN t.name`, then dedups, drops all-digit names, and sorts alphabetically. Sort with plain `str` ordering first, for deterministic output. 5. `mod.rs`: add `pub fn load_used_names(base_folder: &Path, tz: Tz) -> Result<Vec<String>, StreamdError>`. Steps: 1. `store::open_or_rebuild` 2. `sync::sync_structural` 3. `build_projection(canonical_configuration(), tz)` if `!projection_is_fresh` 4. `query::used_tag_and_marker_names` It returns `Err` when the cache is locked or unusable, and callers fall back. ### 2. CLI command `streamd tags` - `src/cli/args.rs`: add `Commands::Tags` with the doc comment "List all tags and markers used in the stream, alphabetically". - `src/cli/commands/tags.rs` (new; register it in `commands/mod.rs` and dispatch it in `main.rs`/`cli/mod.rs` the same way `Cache` is): - `run()` loads `Settings`, then calls `run_in(base_folder)`. `run_in` takes the timezone from `load_repository_config(base_folder)?.timezone_or_utc()`, same as `cache rebuild`. - `collect_used_names(base_folder, tz) -> Result<Vec<String>, StreamdError>`: - Use `cache::load_used_names` unless `cache::is_disabled()`. - On `Err`, or when the cache is disabled, **fall back** to a direct recursive scan. Use `cache::sync::scan_disk` for the paths, then `parse_markdown_file`, and collect `markers` + `tags` from the parsed `Shard` tree (no localization needed). Apply the same all-digit filter, dedup and sort. - Put the filter/dedup/sort in one shared helper (e.g. `normalize_used_names(impl Iterator<Item=String>)`) so both paths produce identical output. - Output: one name per line, prefixed with `@`, to stdout. Print nothing for an empty corpus. ### 3. LSP completion (`src/cli/commands/lsp.rs`) 1. `LspState`: add `used_names: std::sync::RwLock<Vec<String>>`. 2. `LspState::refresh_used_names(&self)`: - Try `cache::load_used_names(&self.base_folder, self.tz)`, unless the cache is disabled. - On failure, fall back to collecting `markers` + `tags` from every entry currently in `file_cache` (walk the `LocalizedShard` trees), normalized with the same helper. Either way no disk reads happen on this path, and it never fails. - Store the result in `used_names`. 3. Call sites: - `build_state`, after `seed_from_cache()` (covers start-up and `.streamd.toml` reload). - `did_save`. - The markdown branch of `did_change_watched_files`. Run the refresh via `tokio::task::spawn_blocking` (clone the `Arc<LspState>`) so the graph rebuild never blocks request handling. A completion request that arrives meanwhile just sees the previous set. 4. `completions_for_line`: add a parameter `used_names: &[String]`, keeping the function pure and unit-testable. After the config markers, add an item for each used name that: - matches the typed prefix, case-insensitively, like today; - is **not** already in `config.markers` (no duplicate labels). Each item gets `label: "@name"`, `insert_text: name`, `kind: CompletionItemKind::TEXT`, `detail: "Used in stream"` and `sort_text: "1b_{name}"`. Resulting order: conditional config markers (`0_`), then config markers (`1_`), then used names (`1b_`), then date/time snippets (`2_`). 5. The `completion` handler passes `&state.used_names.read()`. ### 4. Tests - `cache/projection.rs` / `cache/query.rs`: - Markers, including unconfigured ones like `@Project-X`, appear in `used_tag_and_marker_names`. - A name used as both marker and tag is listed once. - All-digit names are excluded and output is sorted. - `tags_co_occurring_with` is unaffected by `MARKED` edges. - Staleness: - Build the projection, then run `sync_structural` after editing or adding a file: `projection_is_fresh` must be `false`. - After `load_used_names`, the new tag is returned. - An unchanged corpus does **not** rebuild the projection. Check that the dirty flag stays false and the fingerprint is unchanged. - `cli/commands/tags.rs`: - Cached and uncached (`STREAMD_NO_CACHE=1` / locked cache) paths return identical results on a corpus with nested files, unconfigured markers, content tags and a temporal marker. - Nested files are included. - `lsp.rs`: - `completions_for_line` suggests a used tag matching the prefix. - It does not duplicate a used name that is also a config marker. - Used names sort after config markers and before the date/time snippets. - Existing tests keep passing, updated for the new parameter (pass `&[]`). - `cache/mod.rs`: `load_used_names` on a fresh cache builds the projection and returns names. ### 5. Docs - `REQUIREMENTS.md`: - Add `streamd tags` to R20. - Add a short R-section for it, covering scope (recursive, markers + tags, digits excluded, alphabetical, `@`-prefixed, cache with direct-scan fallback). - Extend R25a with used-name completions (source, dedup against config, sort order, refresh cadence). - Update R26a: `MARKED` edge, `projection_dirty`, auto-rebuild when stale for these consumers, schema v2. - `README.md`: - Add `streamd tags` to the Commands list. - Update the LSP feature table row for `@` completions to mention used tags/markers. - Change the "no CLI surface is exposed for them yet" sentence to mention `streamd tags`. ### 6. Verification - `cargo test`, `cargo clippy`, `cargo fmt`, then `nix flake check`. - Manual check: run `streamd tags` on a sample stream. Start `streamd lsp` in an editor, type `@` followed by a prefix of a tag used only in another file, and confirm it is suggested. Add a new tag, save, and confirm it becomes suggestible.

kfickel added the

planned

label 2026-09-14 13:23:02 +02:00

kfickel referenced this issue from a pull request that will close it, 2026-09-14 13:42:54 +02:00

feat: list and complete used tags and markers (v0.7.0) #150

kfickel commented 2026-09-14 13:43:09 +02:00

Author

Owner

Copy link

Implementation notes

Implemented in PR #150 (branch 149_used-tags-markers-completion, released as v0.7.0).

• Duration: about 18 minutes (2026-09-14 13:24 – 13:42 CEST), from starting the implementation to pushing. The earlier refinement isn't included.
• Tokens: about 162k, read from the session's token counter. That figure counts the whole context, not only newly generated text.
• Commits: 8, tests written before each piece of code.

Findings during implementation

1. The projection was almost never current. Before this change, only streamd cache rebuild built it, and ordinary commands synced files without updating it. A dirty flag stored in the cache now tracks that. A flag on one sync's result wouldn't be enough, because another command may already have synced the changed files.
2. No schema bump needed. The plan proposed bumping to v2. Instead, a projection built before this change has no dirty flag, counts as stale and gets rebuilt, so users' caches aren't thrown away.
3. Flaky tests, including the failing Nix/home-manager build of v0.6.0. Two existing tests changed STREAMD_CACHE_DIR and STREAMD_NO_CACHE process-wide while the suite ran in parallel. That broke test_cache_lives_outside_the_stream_folder in the Nix build and occasionally a new LSP test. They now pass the value in directly, and no test changes the environment. 20 full runs in a row passed.
4. The LSP fallback was too narrow. When the cache was unusable, it only saw files the server had already parsed. It now reads the workspace files and reuses parsed buffers.
5. Files without a date in their name are skipped on both paths. The cache skips them (e.g. README.md), so the direct scan skips them too, and a test checks that both print the same list.
6. Follow-up: STREAMD_BASE_FOLDER (R23) isn't implemented. Settings::load never reads it. A first smoke test relied on it and read the real stream folder by accident (read-only).
7. Follow-up: tags keep trailing punctuation. Real data shows near-duplicates like @Idea / @Idea: and @TNG / @TNG?, because the tag pattern (R1) includes punctuation. Fixing that belongs in extraction.

## Implementation notes Implemented in PR #150 (branch `149_used-tags-markers-completion`, released as v0.7.0). - **Duration:** about 18 minutes (2026-09-14 13:24 – 13:42 CEST), from starting the implementation to pushing. The earlier refinement isn't included. - **Tokens:** about 162k, read from the session's token counter. That figure counts the whole context, not only newly generated text. - **Commits:** 8, tests written before each piece of code. ### Findings during implementation 1. **The projection was almost never current.** Before this change, only `streamd cache rebuild` built it, and ordinary commands synced files without updating it. A dirty flag stored in the cache now tracks that. A flag on one sync's result wouldn't be enough, because another command may already have synced the changed files. 2. **No schema bump needed.** The plan proposed bumping to v2. Instead, a projection built before this change has no dirty flag, counts as stale and gets rebuilt, so users' caches aren't thrown away. 3. **Flaky tests, including the failing Nix/home-manager build of v0.6.0.** Two existing tests changed `STREAMD_CACHE_DIR` and `STREAMD_NO_CACHE` process-wide while the suite ran in parallel. That broke `test_cache_lives_outside_the_stream_folder` in the Nix build and occasionally a new LSP test. They now pass the value in directly, and no test changes the environment. 20 full runs in a row passed. 4. **The LSP fallback was too narrow.** When the cache was unusable, it only saw files the server had already parsed. It now reads the workspace files and reuses parsed buffers. 5. **Files without a date in their name are skipped on both paths.** The cache skips them (e.g. README.md), so the direct scan skips them too, and a test checks that both print the same list. 6. **Follow-up: `STREAMD_BASE_FOLDER` (R23) isn't implemented.** `Settings::load` never reads it. A first smoke test relied on it and read the real stream folder by accident (read-only). 7. **Follow-up: tags keep trailing punctuation.** Real data shows near-duplicates like `@Idea` / `@Idea:` and `@TNG` / `@TNG?`, because the tag pattern (R1) includes punctuation. Fixing that belongs in extraction.

kfickel closed this issue 2026-09-14 13:55:39 +02:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

renovate/lock-file-maintenance

v0.9.0

v0.8.0

v0.7.0

v0.6.0

v0.5.0

v0.4.0

v0.3.0

v0.2.7

v0.2.6

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.1

Labels

Clear labels   

planned

The feature is ready to be implemented by an AI agent

No labels

planned

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

kfickel/streamd#149

No description provided.