Performance tuning

Fit Memtrace to your machine: CPU governor tradeoffs, auto-tuned host tiers, RSS guardrails, batch sizes, rerank, embed cache in CI, and diagnosing slow indexing.

Memtrace auto-tunes embedding batch size, ONNX thread count, and RSS guardrails from detected RAM, CPU, and accelerator signals. Most users never touch these knobs — this page is for when defaults are not right for your host or workload.

Overview#

Term	Meaning
ONNX runtime	Local inference engine for embedding and rerank models. No network unless you use a remote embedder.
Intra-op threads	CPU threads ONNX uses per operation. More threads = faster ops but more RAM per batch.
Batch size	How many symbols Memtrace embeds at once. Memory scales roughly linearly with batch size.
RSS	Resident Set Size — physical RAM the process holds. The embed loop yields when RSS crosses the guardrail.
CoreML / ANE	Apple model execution provider. The verified 1.1.13 nightly requests CoreML by default on Apple Silicon. First use can incur compilation; MEMTRACE_DISABLE_COREML=1 forces CPU-only execution.

For embedding provider choice (local vs remote), see Embedding providers. For every env var, see Environment variables.

CPU governor#

Memtrace's CPU governor manages background CPU allocation to support foreground responsiveness on shared systems. The policy adapts to system load, battery power, user activity and memory pressure, and permits additional background work during idle periods. It is opt-in: set MEMTRACE_CPU_GOVERNOR=on in the daemon's launch environment and restart to enable it.

Background pacing can extend indexing completion time while reserving resources for concurrent applications. The CPU governor reference covers configuration, Windows/Linux/macOS support and diagnostics. Workload-specific measurements are documented separately in the validation reference.

How auto-tuning works#

On startup Memtrace scores your host (0–11) from RAM, CPU core count (with Apple Silicon tier table), and accelerator presence, then picks a tier:

Score	Tier	embed_intra_op_threads	embed_batch_size	embed_rss_limit_gb
0–2	light	1	8	3–4
3–6	standard	2	16	6
7+	heavy	4	64	10–20

The daemon banner prints the resolved profile, for example: Host profile: Apple M3 Pro · 12 (6P+6E) · 18 GB · score=5 · tier=standard · embed=int8. Override the tier with MEMTRACE_TIER=light|standard|heavy, or set individual knobs from Environment variables.

V0.4.60 MEMORY BASELINE

Defaults moved in v0.4.60: median cold-reindex peak RSS dropped ~15% with variance collapsing from ~29% spread to within ~1% of median on repeated runs. mimalloc is the default allocator; a unified hot cache (MEMTRACE_UNIFIED_CACHE_MB, default 256 MB) replaced several independent caches. Retrieval accuracy (acc@1) stayed bit-identical on the 1k-query bench.

Common scenarios#

16 GB Apple Silicon — high RSS during indexing

Pre-v0.3.31 builds could spike past 27 GB resident during indexing. v0.3.31+ caps ORT intra-op threads to 2, embed batch to 16, and adds a 6 GB RSS guardrail with back-pressure. If RSS is still high:

terminal

export MEMTRACE_TIER=light
export MEMTRACE_EMBED_BATCH_SIZE=4
export MEMTRACE_EMBED_INTRA_OP_THREADS=1
export MEMTRACE_EMBED_RSS_LIMIT_GB=4
export MEMTRACE_RERANK=off
memtrace stop && memtrace start

Watch for embed: RSS sample batch_idx=… rss_mb=… limit_mb=… in daemon logs — staying under the limit means the guardrail is working.

8 GB laptop — even bge-small is tight

terminal

export MEMTRACE_TIER=light
export MEMTRACE_EMBED_MODEL=bge-small
export MEMTRACE_VECTOR_DIMS=384
export MEMTRACE_EMBED_QUANT=int8
export MEMTRACE_EMBED_BATCH_SIZE=4
export MEMTRACE_EMBED_INTRA_OP_THREADS=1
export MEMTRACE_RERANK=off
export MEMTRACE_DISABLE_COREML=1
memtrace start

Expect ~6 pts lower acc@1 vs default jina-code — intentional tradeoff for 4 GB-class hosts.

64 GB workstation — maximize throughput

terminal

export MEMTRACE_TIER=heavy
export MEMTRACE_EMBED_BATCH_SIZE=128
export MEMTRACE_EMBED_INTRA_OP_THREADS=8
export MEMTRACE_EMBED_QUANT=fp32
memtrace start

Skip embedding for structural-only passes

Parser and graph work dominate less than embedding on large repos. Skip semantic search entirely:

terminal

export MEMTRACE_SKIP_EMBED=1
memtrace index <path>

Or use a smaller model via memtrace embed set bge-small (~3× faster than jina-code, ~6 pts lower retrieval accuracy).

Faster queries — disable rerank

terminal

export MEMTRACE_RERANK=off
memtrace stop && memtrace start

Typical tradeoff: p50 ~50–150 ms instead of ~450–870 ms, with ~3–4 pp lower acc@1 on agent-style queries.

Repeated CI re-indexing

The on-disk embed cache at ~/.memtrace/embed-cache/ keys by symbol AST hash — unchanged symbols are cache hits even after wiping .memdb/. Mount the cache directory as a CI volume so embed passes become nearly free after the first run.

REMOTE EMBEDDERS SKIP LOCAL RAM PRESSURE

Remote embedding moves compute off your machine — the local RSS gate does not apply to HTTP embedding batches. Useful on constrained laptops and CI runners without GPUs.

Knobs and tradeoffs#

MEMTRACE_EMBED_BATCH_SIZE

Value	RAM	Throughput	Best for
4	~30% lower	~30% slower	RPi / very tight RAM
8	Light default	baseline	8 GB Mac
16	Standard default	baseline	16 GB Mac
64	Heavy default	best	32+ GB workstation
128	extreme	marginal gain	64+ GB GPU box

MEMTRACE_EMBED_INTRA_OP_THREADS

Doubling threads roughly doubles per-op scratch RAM. Above 4 on a non-GPU host rarely helps — embedding is usually memory-bandwidth-bound.

MEMTRACE_RERANK

Setting	acc@1 (agent queries)	p50 latency	Extra RAM
off	~70%	~50 ms	0
on (default)	~74%	~450 ms	~75 MB int8 model

MEMTRACE_EMBED_RSS_LIMIT_GB

Soft ceiling — crossing it yields the embed loop for 50 ms and logs a warning. Do not set so low that it fires constantly; defaults scale with host RAM.

MEMTRACE_UNIFIED_CACHE_MB

In-process hot cache (default 256 MB). Raise to 512 or 1024 on RAM-rich hosts; 0 disables.

MEMTRACE_ORT_LOW_RSS

Disables ORT memory arenas on every ONNX session site — empirical −2.7% RSS for −19% throughput on shipping model sizes. Off by default.

Diagnose without changing anything#

terminal

memtrace status
# data dir, graph counts, noise-filter skips on the running daemon

Daemon logs (foreground memtrace start or log files under ~/.memtrace/):

log lines

ort: global thread pool capped — intra_op=2, inter_op=1
embed: RSS sample batch_idx=32 rss_mb=4892 limit_mb=6144

The dashboard Value tab breaks down per-query latency. For embed provider health: memtrace embed status and memtrace embed test.

NOTE

Trim indexed noise with .memtraceignore before chasing performance — fewer files means faster walks and cheaper embed passes.

See also Troubleshooting for circuit-breaker and ONNX load failures, and memtrace warmup to pre-pay CoreML compile cost.