Project Aether Milestone 1 vault and MCP architecture
Status: implemented (M1, M2 & M3, 2026-11-18). crates/aether-vault (muninn-aether-vault and aether-mcp) and obsidian-aether-plugin are implemented in the workspace.
Problem
Milestone 1 builds a local Obsidian vault daemon and a three-tool stdio MCP server inside a new workspace crate at crates/aether-vault, named muninn-aether-vault. Existing crates in Aether SDK separate storage contracts from graph and fusion ranking. muninn-aether-adapters provides SqliteRecordStore for opaque JSON payloads without FTS5 full-text search, wikilink edge queries, or a self-write hash ledger.
Offline Cargo.lock constraints also shape the build boundary. The lockfile includes rusqlite 0.41 with bundled SQLite and FTS5, sha2 0.11, hex 0.4, walkdir 2.5, tokio, serde_json, chrono, petgraph, thiserror, tracing, and uuid without the v5 feature. Neither notify, rmcp, nor serde_yaml is present in Cargo.lock. The crate must parse Obsidian notes, exclude private notes at the boundary, fuse FTS5, local vectors, and PersonalizedPageRank scores into deterministic HOT prompt prefixes, reconcile Aether/Inbox.md approvals without watcher loops, and expose the three MCP tools without leaking internal database or JSON-RPC types.
Usage
Consumers depend on muninn-aether-vault and interact with VaultEngine, McpServer, and validated domain types.
Quickstart daemon and MCP server
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use muninn_aether_core::Result;
use muninn_aether_vault::{McpServer, VaultConfig, VaultEngine};
#[tokio::main]
async fn main() -> Result<()> {
let vault_root = PathBuf::from(std::env::args().nth(1).expect("vault path"));
let config = VaultConfig::new(&vault_root)
.with_watch_interval(Some(Duration::from_secs(2)));
let engine = Arc::new(VaultEngine::open(config).await?);
let server = McpServer::new(Arc::clone(&engine));
server.serve_stdio().await?;
engine.shutdown().await
}
Call site 1. Hybrid recall with deterministic prefix caching
use muninn_aether_vault::{RecallRequest, ScopeTag, TokenBudget, VaultConfig, VaultEngine};
async fn build_context(vault_root: &std::path::Path) -> muninn_aether_core::Result<String> {
let engine = VaultEngine::open(VaultConfig::in_memory(vault_root)).await?;
engine.sync_vault().await?;
let mut req = RecallRequest::new("SQLite write locking and WAL transactions");
req.scope = Some(ScopeTag::new("engineering")?);
req.max_tokens = TokenBudget::new(1500)?;
req.include_hot_profile = true;
let response = engine.recall(req).await?;
assert!(response.tokens_used <= 1500);
let _stable_hash = response.prefix_hash;
Ok(response.formatted_context)
}
Call site 2. Non-destructive staging into the Inbox
use muninn_aether_vault::{StageMemoryRequest, VaultConfig, VaultEngine, VaultTier};
async fn propose_fact(engine: &VaultEngine) -> muninn_aether_core::Result<()> {
let candidate_id = engine
.stage_memory(
StageMemoryRequest::new(
"Use TransactionBehavior::Immediate for all SQLite writes.",
VaultTier::Warm,
)
.with_scope("engineering")?
.with_target_note("Architecture/Storage.md"),
)
.await?;
assert!(candidate_id.as_str().starts_with("cand_"));
Ok(())
}
Call site 3. Human approval and verbatim source inspection
use muninn_aether_core::AetherError;
use muninn_aether_vault::{SourcePointer, VaultEngine};
async fn approve_and_verify(engine: &VaultEngine) -> muninn_aether_core::Result<String> {
let report = engine.sync_vault().await?;
assert!(report.inbox_approved <= report.files_scanned + report.inbox_approved);
let ptr = SourcePointer::parse("Architecture/Storage.md#WAL-Mode")?;
let verbatim = engine.inspect_source(ptr.as_str()).await?;
assert!(matches!(
SourcePointer::parse("../../etc/passwd"),
Err(AetherError::InvalidInput(_))
));
Ok(verbatim)
}
Shape
The design places one deep owner, VaultEngine, over an internal SQLite WAL store, pure Markdown parser, deterministic prefix formatter, and atomic Inbox reconciler. Public callers see VaultEngine, McpServer, and validated request and response types. Storage schemas, rusqlite connections, RawFrontmatterTier, and JSON-RPC wire structs stay private to the crate.
Module map
crates/aether-vault/
├── Cargo.toml
└── src/
├── lib.rs # Public re-exports only
├── types.rs # Branded primitives, AMU model, request and response types
├── parser.rs # IgnoreRules, YAML frontmatter, ATX splitter, Embedder trait
├── inbox.rs # Atomic writes and PRD Section 7.3 Inbox parser and renderer
├── prefix.rs # Deterministic HOT prefix builder and dynamic token packer
├── store.rs # Internal SQLite WAL, FTS5, graph edges, and self-write ledger
├── engine.rs # VaultEngine orchestrator and background polling loop
├── mcp.rs # Thin JSON-RPC 2.0 MCP handler and stdio stream loop
└── bin/
└── aether-mcp.rs # CLI daemon entrypoint
Core data structures and boundary invariants
SourcePointerandVaultPath(types.rs). Both normalize backslashes to forward slashes, strip leading./segments, and reject empty paths, leading slashes, NUL bytes,..traversal segments, and hidden system directories such as.obsidian/,.git/, and.trash/(per principle-boundary-discipline).AmuId(types.rs). Derives a 128-bit UUID from the SHA-256 digest of the canonicalSourcePointerstamped with RFC 4122 version and variant bits, mapping one-to-one tomuninn_aether_core::MemoryIdwithout requiring theuuidv5feature (per principle-type-system-discipline).CandidateId,ContentHash,ScopeTag, andTokenBudget(types.rs). Each validates its format (cand_with 12 lowercase hex characters, 64 lowercase hex characters, non-empty lowercase scope label, and positive token ceiling) at construction time.VaultTierandRawFrontmatterTier(types.rs,parser.rs). PublicVaultTiercontains onlyHot,Warm, andCold.Privateexists only on the crate-privateRawFrontmatterTierinsideparser.rs, so private notes cannot enterStageMemoryRequest, SQLite storage, or recall results (per principle-type-system-discipline).AmuandAmuStatus(types.rs). Capturesamu_id,tier,kind,status,distilled_fact,source_pointer,graph_edges,valid_from,superseded_by,scopes,authority_weight, andcontent_hash.
Ingestion, concurrency, and recall flow
- Parsing and exclusion (
parser.rs).IgnoreRulesloads.aetherignoreonce and enforces one shared list of built-in exclusions across directory scans andinspect_sourcechecks.parse_noteextracts YAML frontmatter, returnsParsedNote::Privateimmediately whenaether_tierisprivate, splits body lines at ATX headings outside fenced code blocks, emits#_preamblewhen text precedes the first heading, extracts[[wikilinks]]and#tags, and scoresauthority_weightwith evergreen notes aboveDailyorJournalnotes and unchecked task lists. - Storage and loop prevention (
store.rs,engine.rs).VaultStoreopens one SQLite WAL connection wrapped inArc<Mutex<Option<Connection>>>and runs queries throughtokio::task::spawn_blockingwithTransactionBehavior::Immediatefor writes (per principle-separate-before-serializing-shared-state).sync_vault,handle_fs_event,stage_memory, andreconcile_inboxserialize vault file mutations underwrite_gatewhile leavingrecallandinspect_sourcelock-free under WAL snapshots. Every atomic write computes itsContentHash, inserts(rel_path, hash)intoself_write_ledgerbeforerename, and skips matching self-originated hashes on later scans while pruning stale ledger entries (per principle-make-operations-idempotent). - Hybrid retrieval and prefix packing (
engine.rs,prefix.rs).VaultEngine::recallsupports empty queries wheninclude_hot_profileistrueby returning only the staticHOTprefix. For non-empty queries,VaultEnginequeries SQLite FTS5 BM25, scores active rows with local 256-dim vectors fromHashedNgramEmbedderviaDistanceMetric::Cosine, builds a deduplicatedSubgraphforPersonalizedPageRankandgraph_scores, and fuses all three ranked lists withmuninn_aether_cortex::recall::rrfscaled byauthority_weight.prefix.rssortsHOTitems strictly byamu_idascending inside a 2,000-token ceiling with zero volatile timestamps, computesprefix_hash, and appends dynamicWARMandCOLDitems withinmax_tokens. - Inbox staging and MCP server (
inbox.rs,mcp.rs).stage_memoryformats proposals in the PRD Section 7.3 Markdown checkbox shape and replacesAether/Inbox.mdvia temp-file write,sync_all, andrename.reconcile_inboxcompiles checked[x]items into active SQLite AMUs, marks anysupersedes_amu_idtarget asSuperseded, drops[-]items, and rewrites the remaining pending items atomically.McpServerdispatchesinitialize,ping, andtools/listplustools/callforaether_recall,aether_stage_memory, andaether_inspect_source.
Interface depth verdict
Callers learn two primary types (VaultEngine and McpServer) and small value objects. Behind that boundary sit file walking, .aetherignore matching, YAML and Markdown parsing, SQLite WAL storage with FTS5, feature-hash vector scoring, PageRank graph ranking, RRF fusion, canonical prefix caching, and atomic Inbox reconciliation (per principle-minimize-reader-load). Every MCP tool call completes within two internal hops.
Synthesis decision
Candidate 1 (VaultEngine single-owner design) won the Phase C evaluation with a score of 23 out of 25 against Candidate 2's score of 16 out of 25, and Candidate 1 becomes the base architecture. Candidate 2 split state across records.db via SqliteRecordStore and vault.db, which hit two structural conflicts: MemoryScope::index_key only represents "project" and "user:<id>" rather than vault topic scopes, and SqliteRecordStore::mark_indexed deletes tombstones while vault.db keeps superseded metadata. Candidate 1 avoids split ownership by committing AMUs, FTS5 rows, graph edges, self-write hashes, and Inbox state inside one SQLite WAL transaction.
Three parts from Candidate 2 were grafted into the Candidate 1 base:
- Internal
Embeddertrait withHashedNgramEmbedder. Replaced Candidate 1's bare free function with apub(crate) trait Embedderseam so Milestone 2 neural embeddings can replace local 256-dim trigram vectors without changingVaultEnginecallers. - Pre-heading
#_preamblesplitting and self-write consumption. Preserved notes that start with prose before any ATX heading, and added consumption and pruning onself_write_ledgerso reverting a file to an older daemon hash is never skipped forever. - Empty-query
HOTprofile recall. Allowedrecallwith an empty query wheninclude_hot_profileistrueso callers can assemble static system prompt prefixes on cold start.
Synthesis cleanup also branded ContentHash as a 64-hex newtype, standardized CandidateId, kept one IgnoreRules exclusion list shared across scan and inspect paths, and avoided storing redundant verbatim section copies in SQLite when inspect_source reads live Markdown from disk.
Tradeoffs accepted
- We accept a crate-local SQLite WAL schema inside
muninn-aether-vaultin exchange for single-transaction ACID consistency across AMUs, FTS5 indexes, wikilink edges,self_write_ledger, andstaged_candidates. - We accept 256-dimensional signed trigram feature hashing as the default Milestone 1
Embedderin exchange for zero network or model dependencies, preserving an internal trait boundary for Milestone 2 dense embeddings. - We accept
walkdirplus SHA-256 polling instead of OS kernel watchers in exchange for strict offline compatibility withCargo.lock, becausenotifyis not cached inCargo.lock. - We accept re-reading the target Markdown file on disk during
aether_inspect_sourcein exchange for immediate live accuracy and zero storage ofprivatenote bodies, keepingamusrows lightweight.
Alternatives considered
- Dual-database event pipeline over
SqliteRecordStoreandcortex::Indexer(Candidate 2). Exposes callers to two SQLite files, requires translatingScopeTagonto incompatibleMemoryScopekeys, and splits ownership betweenrecordstombstones andamu_metastatus fields. Lost on interface depth and single-owner consistency. - Multi-actor channel pipeline (
Scanner,Parser,Indexer,Inboxactors). Organizes modules by execution phase instead of owned domain state, deepens call chains past three hops, and fragments crash recovery across channels. Lost due to temporal decomposition and split ownership.
Implementation reconciliation
VaultEngineexposes bothsync_vaultandhandle_fs_eventplusreconcile_inboxso file-watcher loop tests and Inbox approval tests can drive single-event and full-scan transitions directly while keeping internal store state private.
Open questions and risks
- Should approved
[x]candidates without an explicittarget_noteremain solely in the compiled SQLite index or also append toAether/Compiled/Approved.mdsoaether_inspect_sourcecan cite an on-disk Markdown anchor? - Should Milestone 2 trigger automatic re-embedding on startup when
Embedder::dimensions()differs from the vector blob size stored in SQLite?
Next implementation step
Add crates/aether-vault to root Cargo.toml and write failing Red TDD tests for types.rs covering SourcePointer normalization, path traversal rejection, AmuId deterministic MemoryId mapping, and ContentHash validation.
Milestone 3 Addendum (Team Git Sync, Scoped Auth & KV Warming)
Milestone 3 extends crates/aether-vault to SQLite schema version 3 (VAULT_SCHEMA_VERSION = 3) inside the same single-owner VaultEngine boundary.
auth.rs& SQLiteagent_api_keys. AddsAgentId,KeyId,ApiKeySecret(aek_<hex>),AgentRole(Reader < Contributor < Admin),ScopedApiKey, and constant-time domain-separated SHA-256 verification.recall_as,stage_memory_as,inspect_source_as,pending_inbox_as, andwarm_kv_cache_asintersect agent scopes before top-K truncation and auto-stampproposed_by.git.rs& Monotonic Lattice. AddsVaultEngine::git_syncwith.gitignore.aether/protection,redact_url_userinfosanitization, conflict-marker skipping for human notes, and deterministicApproved > Rejected > Pendingconflict resolution forAether/Inbox.mdandAether/Compiled/Approved.md.kv.rs& SQLitekv_cache_blocks. AddsKvCacheAdapter(InMemoryKvCacheAdapter,VllmLmCacheHttpAdapterfor vLLM/SGLang + LMCache over HTTP with offline fallback),KvBlockHash(PagedAttention chained prefix blocks and CacheBlend AMU chunks), tiered LRU demotion (GpuHbm,HostDram,NvmeSsd),RecallResponse::kv_hint, and transactional eviction whenever AMUs change or supersede.- Transport & CLI Extensions (
http.rs,mcp.rs,aether-mcp.rs,obsidian-aether-plugin). AddsControlPlaneServer::bind_with_auth(/v1/recall,/v1/stage,/v1/inspect,/v1/mcp,/v1/git/sync,/v1/kv/warm,/v1/kv/stats,/v1/auth/keys,/v1/auth/keys/revoke),McpServer::with_api_keywithkv_modelandagent.default_write_scope(), CLI flags (--require-auth,--bootstrap-admin-key,--api-key,--kv-endpoint,--kv-model), and remote HTTPS + bearer auth commands inobsidian-aether-plugin.