Observability & OpenTelemetry Traces
Aether SDK emits structured tracing spans across muninn-aether-sdk, muninn-aether-cortex, and muninn-aether-adapters out of the box so distributed traces connect end-to-end with zero per-call boilerplate.
1. Zero-Boilerplate Trace Hierarchy
MemoryManager::builder(record_store) wraps configured stores in Traced<T> by default (with_connector_tracing(true)). Every operation produces a connected parent-child span tree:
aether.recall (MemoryManager) [project_id, mode, consistency, top_k, otel.status_code]
├── aether.cortex.vector_search [project_id, top_k]
├── aether.cortex.graph_search [project_id, seed_count, hops]
├── aether.cortex.ppr (PersonalizedPageRank)
└── connector_op (Traced<T>) [otel.kind="client", db.system, connector, project_id, op, otel.status_code]
And for writes and background sync (remember, update, forget, flush):
aether.remember (MemoryManager) [project_id, scope, durability, otel.status_code]
├── connector_op (Traced<RecordStore>) [op="record_create", otel.kind="client", db.system, otel.status_code]
└── aether.cortex.index_record (Indexer) [project_id, memory_id, tombstone]
└── connector_op (Traced<Vector/Graph/Rec>) [op="upsert_batch" | "upsert_relations" | "record_mark_indexed"]
aether.flush (MemoryManager) [otel.status_code]
└── aether.cortex.index_pass (Indexer) [indexed, failed]
└── aether.cortex.index_record [project_id, memory_id, tombstone]
Span & Event Details
- Root spans (
muninn-aether-sdk):aether.remember,aether.recall,aether.update,aether.forget,aether.flush,aether.shutdownrecordotel.status_code("OK"or"ERROR") and emittracing::error!on failure. - Orchestration spans (
muninn-aether-cortex):aether.cortex.index_pass,aether.cortex.index_record,aether.cortex.vector_search,aether.cortex.graph_search,aether.cortex.ppr. - Connector client spans (
muninn-aether-adapters::Traced):connector_opwithotel.kind = "client",db.system,connector,project_id,op(record_create,record_get,record_get_many,record_cas_update,record_tombstone,record_pending,record_pending_for,record_session_items,record_mark_indexed,initialize,describe,upsert_batch,delete,search,upsert_entities,upsert_relations,retract_memory_facts,neighborhood), andotel.status_code. - Resilience events (
muninn-aether-adapters):Resilient<T>emitstracing::warn!on transient retry attempts;Timeout<T>emitstracing::warn!on timeout.
Opting Out of Auto-Wrapping
let manager = MemoryManager::builder(record_store)
.vector(vector_store)
.graph(graph_store)
.with_connector_tracing(false)
.build();
2. One-Line OpenTelemetry Setup (feature = "otel")
Enable the otel feature on muninn-aether-sdk to use the built-in OTLP gRPC preset:
[dependencies]
muninn-aether-sdk = { version = "0.1.0", features = ["otel"] }
In your application entrypoint:
use muninn_aether_sdk::telemetry::{init_otel, LogFormat, OtelConfig};
use muninn_aether_sdk::MemoryManager;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Reads OTEL_EXPORTER_OTLP_ENDPOINT and RUST_LOG from the environment.
// Dropping `_telemetry` (or calling `_telemetry.shutdown()`) flushes pending spans & logs.
let _telemetry = init_otel(
OtelConfig::from_env("my-agent-service")
.with_log_format(LogFormat::Pretty),
)?;
let manager = MemoryManager::builder(record_store)
.vector(vector_store)
.graph(graph_store)
.build();
Ok(())
}
Where Logs & Traces Go
- OTLP Traces: Exported via gRPC to
OTEL_EXPORTER_OTLP_ENDPOINT(defaulthttp://localhost:4317). - OTLP Logs: Exported via gRPC (
opentelemetry-appender-tracing) to the same OTLP endpoint whenotlp_logs = true(the default), automatically correlated withtrace_idandspan_id. Toggle with.with_otlp_logs(false). - Console Logs: Printed to
stdoutviatracing_subscriber::fmt(LogFormat::Pretty,LogFormat::Json, orLogFormat::None), filtered byRUST_LOG(defaultinfo).
3. Testing OpenTelemetry Locally
Spin up a local all-in-one Grafana LGTM container (OTel Collector + Tempo + Loki + Grafana UI):
docker run --rm -p 3000:3000 -p 4317:4317 -p 4318:4318 \
grafana/otel-lgtm:latest
Then run your app or the included demo (crates/aether-sdk/examples/otel_demo.rs) with OTEL_EXPORTER_OTLP_ENDPOINT:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
RUST_LOG=info \
cargo run -p muninn-aether-sdk --example otel_demo --features otel
Bringing Your Own Subscriber
init_otel uses tracing_subscriber::registry().try_init(). If your host application already installed a global subscriber / OpenTelemetry pipeline, tracing spans flow into it automatically and calling init_otel remains idempotent without panicking.