Embedding providers

Local ONNX presets or BYO remote embedding (OpenAI, Voyage, Ollama, TEI): switch providers, Matryoshka dims, cost and privacy tradeoffs, and the dim-mismatch reset flow.

Overview#

Memtrace embeds every indexed symbol so semantic search (find_code) can match by meaning, not just keywords. You have three choices for where that work happens:

1. Local (default) — ONNX on your machine. Free, private, fast once warm. Uses RAM.
2. Remote, hosted API — HTTP POST to OpenAI, Voyage, or any OpenAI-compatible embeddings endpoint. Costs credits; zero local compute.
3. Remote, self-hosted — Same wire format pointed at Ollama, Infinity, TEI, vLLM, or LM Studio on your LAN. Free and private if you already run inference infra.

All three drive the same indexing and search pipelines. Switch with memtrace embed set — a dim change triggers a guarded MemDB reset rather than silently corrupting the HNSW index.

Config precedence: workspace .memtrace/embed.toml > ~/.memtrace/config.toml > env vars > built-in default (local jina-code, 768d). See memtrace embed for the full command reference.

Quick decision guide#

Local presets#

Four local presets are built in. memtrace embed set <preset> always sets provider=local and clears any remote configuration.

The first embedding run downloads the active model into ~/.memtrace/fastembed_cache and prints an explicit "First run — downloading embedding model" notice so the wait isn't mistaken for a hang. Local presets never leave the machine and never depend on your Memtrace session — an auth failure can never stop local embedding.

$ memtrace embed list
# See all four presets and which config is currently active.

$ memtrace embed set bge-small
# Switch to the smaller/faster 384d local preset.
# Prompts [r]eset/[c]ancel if your MemDB was built at a different dim.

Remote embedding#

memtrace embed set --remote openai-compat is the only remote kind. It speaks the standard POST /embeddings OpenAI-compatible spec — same request and response shape across hosted and self-hosted providers.

# Hosted: OpenAI
export OPENAI_API_KEY='sk-...'   # in ~/.zshrc — never written to config
memtrace embed set --remote openai-compat \
  --url https://api.openai.com/v1 \
  --model text-embedding-3-small \
  --api-key-env OPENAI_API_KEY

# Hosted: Voyage (code-tuned)
export VOYAGE_API_KEY='vk-...'
memtrace embed set --remote openai-compat \
  --url https://api.voyageai.com/v1 \
  --model voyage-code-3 \
  --api-key-env VOYAGE_API_KEY

# Self-hosted: Ollama (no auth)
memtrace embed set --remote openai-compat \
  --url http://localhost:11434/v1 \
  --model nomic-embed-text

Only the name of the API-key env var is stored in TOML — the secret is read at request time and never persisted. Remote auth is entirely separate from your Memtrace session token.

Without --dim, embed set probes <url>/embeddings with a one-token request and records the returned vector length. Verify anytime with memtrace embed test and memtrace embed status — check memdb_dim matches dim.

Provider reference

Matryoshka truncation

OpenAI text-embedding-3-* and Voyage voyage-code-3 / voyage-4-* support smaller output dims — trade a small recall hit for less HNSW memory and faster search. Pass --dim N on embed set:

memtrace embed set --remote openai-compat \
  --url https://api.openai.com/v1 \
  --model text-embedding-3-small \
  --api-key-env OPENAI_API_KEY --dim 512

memtrace embed set --remote openai-compat \
  --url https://api.voyageai.com/v1 \
  --model voyage-code-3 \
  --api-key-env VOYAGE_API_KEY --dim 256

Indexing vs query embedding

Cost and latency

Reference: indexing ~1,500 embeddable symbols (MemFleet-scale repo).

Per-query find_code latency (single query, warm): local jina-code 10–50 ms; Voyage US from Europe 300–500 ms; LAN Ollama 5–30 ms.

Privacy

• Local — symbol bodies never leave your machine.
• Remote hosted — Memtrace sends code-symbol bodies (up to ~1,500 chars each) and query strings over HTTPS. The provider's retention policy applies — read it before using on confidential codebases.
• Self-hosted remote — traffic stays on your hardware; useful on air-gapped workstations that still want HTTP-based model swapping.

Failure modes

After a trip: memtrace embed reset-breaker or restart the daemon. See Troubleshooting for the embed circuit breaker section.

Changing dimensions#

A MemDB HNSW index cannot mix vector dimensions. Every time you switch models or providers, Memtrace compares the new config's dimension against the live MemDB's dimension. If they differ, it will not silently corrupt the index — it walks you through a guarded reset-and-reindex flow instead.

• Interactive (TTY): a prompt offers [r] Reset + reindex now or [c] Cancel (config NOT written).
• Non-interactive (CI, pipelines): the command errors with "dim mismatch detected and stdin is not a TTY" and exits 2 — pass --yes-reset or --no-reset explicitly.
• No MemDB dim known yet: nothing to reconcile — the config is written directly.

Accepting the reset (r or --yes-reset) prints a loud red stderr banner naming the exact directory (! ABOUT TO WIPE MEMDB DATA DIR: <path>), then stops OS supervisors, kills the daemon, and wipes that local MemDB data dir — the exact dir that produced the measured dim, never a re-resolved path from env or workspace anchoring.

$ memtrace embed set bge-small --yes-reset
# Scripts/CI: skips the prompt, wipes the MemDB data dir
# (loud stderr warning shows the exact path), and writes the config.
# Re-run `memtrace start` or `memtrace index .` afterward to reindex.

Hardware acceleration#

Embeddings run through ONNX Runtime. On CPU-only hosts this just works; on GPU-capable hosts Memtrace can accelerate the same models — but every accelerator is opt-in, not default behavior.

Apple Silicon: ANE / CoreML

On macOS, embeddings default to CPU-only even on Apple Silicon. The native CoreML execution provider (which can route work to the Apple Neural Engine) is explicit opt-in via MEMTRACE_ENABLE_COREML=1 — it is not part of the default first-run cost. When enabled, the first inference pays a CoreML graph compile that can take 60–300 seconds cold; run memtrace warmup ahead of time so memtrace start doesn't trip the embed-batch circuit breaker on that spike.

Windows NVIDIA: CUDA / DirectML

On Windows, the engine probes execution providers in order: CUDA → DirectML → CPU. The CUDA execution provider is an optional ~400 MB DLL fetched from a GitHub release asset (not bundled in the npm package) — the installer downloads it automatically when it detects an NVIDIA GPU. Without it, NVIDIA hosts silently run embeddings on DirectML instead of falling all the way back to CPU.

$ memtrace gpu status
# Shows whether the CUDA DLL is installed, whether an NVIDIA GPU is
# detected, and the provider probe order.

$ memtrace gpu install-cuda
# Fetch + sha256-verify the CUDA provider DLL manually (offline installs,
# GPUs added after install, or detection misses). --force re-downloads.

Performance env knobs#

These override the model/dimension resolved from config files for the current process — useful for scripting, one-off runs, or diagnosing a specific host without touching the persistent TOML config.

See also memtrace embed, Performance tuning, memtrace warmup, and Data directories for cache locations.