Home/lazarus
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(ned): asynchronous post-sync plugin system and isolated typesafe_jev plugin #89

Merged
clanker merged 8 commits from feat/ned-post-sync-plugins into main 2026-09-21 19:09:43 +00:00
Conversation 0 Commits 8 Files changed 17 +1487 -11
clanker commented 2026-09-21 18:07:19 +00:00
Member
Copy link

Summary

Architects and implements an optional, generic, asynchronous post-sync plugin subsystem for NED (Notmuch Email Daemon) alongside an isolated TypeSafe System One (Jev) email classification plugin.

Key Changes

  • Generic Post-Sync Plugin Architecture (ned/plugins/):
    • PostSyncContext: Provides plugins with sync outcome stats, client interface, plugin-scoped logger, and an SSE cache invalidation helper (ctx.invalidate("threads")).
    • PostSyncPlugin Protocol: Standardizes the on_post_sync(context) interface while also supporting arbitrary callables.
    • PluginRunner: Runs registered plugins asynchronously in a dedicated FIFO background worker queue after notmuch new and static filter rules complete, ensuring mail sync is never delayed.
    • resolve_plugin: Resolves plugins defined as instances, classes, callables, or import path strings ('module.ClassName' or 'module:ClassName').
  • TypeSafe System One Email Classification Plugin (ned/plugins/typesafe_jev/):
    • Completely isolated from NED core (zero AI dependencies in core). Not enabled by default.
    • Strict Privacy: Evaluates solely sender (from) and subject metadata against the classification criteria. Email message bodies are never read or transmitted.
    • No Hidden Defaults: On first run, auto-generates ~/.config/ned/plugins/typesafe_jev/taxonomy.py containing the full two-facet taxonomy rubrics (Document Type and Domain) and thresholds. The plugin strictly reads this user-owned file; no fallback dictionaries exist in code.
    • Evaluates untagged threads in batches with configurable concurrency and exponential backoff retry.
  • NED Core & Configuration:
    • ned.settings.post_sync_plugins: Setting declaring active post-sync plugins.
    • ned --init-config: Generates a dedicated commented-out Section 4 demonstrating plugin configuration, and automatically scans and comments any detected user plugins in ~/.config/ned/plugins/.
    • ned/sync.py: Dispatches post-sync stats to default_runner after sync completes.
    • ned/daemon.py: Gracefully shuts down default_runner on daemon stop.
  • Documentation:
    • Main README.md: Added 'Post-sync plugins (optional)' section with architecture overview and typesafe_jev configuration.
    • agent.md: Updated project layout, subsystem guide, configuration table, architecture lifecycle, and added Durable Invariant 14 (Post-Sync Plugin Safety & Isolation).
  • Tests:
    • Comprehensive unit test suite in tests/test_ned_plugins.py (23 tests) verifying runner lifecycle, resolution, error resilience, taxonomy file generation, custom rubrics, threshold logic, mocked API payload safety, and config generation.

Verification

pytest test suite passes (551/551 tests passed).

## Summary Architects and implements an optional, generic, asynchronous post-sync plugin subsystem for NED (Notmuch Email Daemon) alongside an isolated TypeSafe System One (Jev) email classification plugin. ### Key Changes - **Generic Post-Sync Plugin Architecture** (`ned/plugins/`): - `PostSyncContext`: Provides plugins with sync outcome stats, client interface, plugin-scoped logger, and an SSE cache invalidation helper (`ctx.invalidate("threads")`). - `PostSyncPlugin` Protocol: Standardizes the `on_post_sync(context)` interface while also supporting arbitrary callables. - `PluginRunner`: Runs registered plugins asynchronously in a dedicated FIFO background worker queue after `notmuch new` and static filter rules complete, ensuring mail sync is never delayed. - `resolve_plugin`: Resolves plugins defined as instances, classes, callables, or import path strings (`'module.ClassName'` or `'module:ClassName'`). - **TypeSafe System One Email Classification Plugin** (`ned/plugins/typesafe_jev/`): - Completely isolated from NED core (zero AI dependencies in core). Not enabled by default. - **Strict Privacy**: Evaluates solely sender (`from`) and `subject` metadata against the classification criteria. Email message bodies are never read or transmitted. - **No Hidden Defaults**: On first run, auto-generates `~/.config/ned/plugins/typesafe_jev/taxonomy.py` containing the full two-facet taxonomy rubrics (Document Type and Domain) and thresholds. The plugin strictly reads this user-owned file; no fallback dictionaries exist in code. - Evaluates untagged threads in batches with configurable concurrency and exponential backoff retry. - **NED Core & Configuration**: - `ned.settings.post_sync_plugins`: Setting declaring active post-sync plugins. - `ned --init-config`: Generates a dedicated commented-out Section 4 demonstrating plugin configuration, and automatically scans and comments any detected user plugins in `~/.config/ned/plugins/`. - `ned/sync.py`: Dispatches post-sync stats to `default_runner` after sync completes. - `ned/daemon.py`: Gracefully shuts down `default_runner` on daemon stop. - **Documentation**: - Main `README.md`: Added 'Post-sync plugins (optional)' section with architecture overview and `typesafe_jev` configuration. - `agent.md`: Updated project layout, subsystem guide, configuration table, architecture lifecycle, and added Durable Invariant 14 (Post-Sync Plugin Safety & Isolation). - **Tests**: - Comprehensive unit test suite in `tests/test_ned_plugins.py` (23 tests) verifying runner lifecycle, resolution, error resilience, taxonomy file generation, custom rubrics, threshold logic, mocked API payload safety, and config generation. ### Verification `pytest` test suite passes (551/551 tests passed).
- Add generic async post-sync plugin infrastructure in ned/plugins (PostSyncContext, PostSyncPlugin, PluginRunner).
- Add isolated TypeSafe System One (Jev) email classification plugin in ned/plugins/typesafe_jev (metadata-only, zero bodies).
- Ensure taxonomy is user-owned via ~/.config/ned/plugins/typesafe_jev/taxonomy.py with no hidden internal defaults.
- Add Section 4 post-sync plugins template and plugin detection to ned --init-config.
- Wire PluginRunner background queue into run_sync and graceful daemon shutdown.
- Update README.md and agent.md documentation.
- Add comprehensive pytest suite in tests/test_ned_plugins.py.
- Add API_KEY = "" (empty template) to ~/.config/ned/plugins/typesafe_jev/taxonomy.py template.
- Load API_KEY in load_taxonomy() and resolve key with priority: constructor arg > taxonomy.py > env var.
- Avoid requiring system environment variables on headless daemon setups like Alpine.
- Add unit tests verifying taxonomy API key loading, template safety, and resolution.
- In tools/typesafe_tagger.py, low confidence emails receive no category tag (and cleans any legacy ai-unassigned tag).
- In TypeSafeJevPlugin and default template, set UNASSIGNED_MARKER to empty and remove unassigned tag addition.
- Update test_typesafe_jev_decide_tags assertions.
- Strip all UNASSIGNED_MARKER references from template.py, plugin.py, and typesafe_tagger.py.
- Low-confidence emails simply receive ai-tagged and no category tags.
- Update test_ned_plugins.py assertions to reflect clean tag decisions.
- Call ensure_taxonomy_file() in __init__ so ~/.config/ned/plugins/typesafe_jev/taxonomy.py is generated on startup when config.py is read.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Home/lazarus!89
No description provided.