Performance tuning

Fit Memtrace to your machine: auto-tuned host tiers, RSS guardrails, batch sizes, rerank tradeoffs, embed cache in CI, and diagnosing slow indexing.

Memtrace auto-tunes embedding batch size, ONNX thread count, and RSS guardrails from detected RAM, CPU, and accelerator signals. Most users never touch these knobs — this page is for when defaults are not right for your host or workload.

Overview#

Term	Meaning
ONNX runtime	Local inference engine for embedding and rerank models. No network unless you use a remote embedder.
Intra-op threads	CPU threads ONNX uses per operation. More threads = faster ops but more RAM per batch.
Batch size	How many symbols Memtrace embeds at once. Memory scales roughly linearly with batch size.
RSS	Resident Set Size — physical RAM the process holds. The embed loop yields when RSS crosses the guardrail.
CoreML / ANE	Apple's on-chip ML accelerator. Opt-in via MEMTRACE_ENABLE_COREML=1; first run pays a graph-compile delay.

For embedding provider choice (local vs remote), see Embedding providers. For every env var, see Environment variables.

How auto-tuning works#

On startup Memtrace scores your host (0–11) from RAM, CPU core count (with Apple Silicon tier table), and accelerator presence, then picks a tier:

Score	Tier	embed_intra_op_threads	embed_batch_size	embed_rss_limit_gb
0–2	light	1	8	3–4
3–6	standard	2	16	6
7+	heavy	4	64	10–20

The daemon banner prints the resolved profile, for example: Host profile: Apple M3 Pro · 12 (6P+6E) · 18 GB · score=5 · tier=standard · embed=int8. Override the tier with MEMTRACE_TIER=light|standard|heavy, or set individual knobs from Environment variables.

V0.4.60 MEMORY BASELINE

Defaults moved in v0.4.60: median cold-reindex peak RSS dropped ~15% with variance collapsing from ~29% spread to within ~1% of median on repeated runs. mimalloc is the default allocator; a unified hot cache (MEMTRACE_UNIFIED_CACHE_MB, default 256 MB) replaced several independent caches. Retrieval accuracy (acc@1) stayed bit-identical on the 1k-query bench.

Common scenarios#

16 GB Apple Silicon — high RSS during indexing

Pre-v0.3.31 builds could spike past 27 GB resident during indexing. v0.3.31+ caps ORT intra-op threads to 2, embed batch to 16, and adds a 6 GB RSS guardrail with back-pressure. If RSS is still high:

terminal

export MEMTRACE_TIER=light
export MEMTRACE_EMBED_BATCH_SIZE=4
export MEMTRACE_EMBED_INTRA_OP_THREADS=1
export MEMTRACE_EMBED_RSS_LIMIT_GB=4
export MEMTRACE_RERANK=off
memtrace stop && memtrace start

Watch for embed: RSS sample batch_idx=… rss_mb=… limit_mb=… in daemon logs — staying under the limit means the guardrail is working.

8 GB laptop — even bge-small is tight

terminal

export MEMTRACE_TIER=light
export MEMTRACE_EMBED_MODEL=bge-small
export MEMTRACE_VECTOR_DIMS=384
export MEMTRACE_EMBED_QUANT=int8
export MEMTRACE_EMBED_BATCH_SIZE=4
export MEMTRACE_EMBED_INTRA_OP_THREADS=1
export MEMTRACE_RERANK=off
export MEMTRACE_DISABLE_COREML=1
memtrace start

Expect ~6 pts lower acc@1 vs default jina-code — intentional tradeoff for 4 GB-class hosts.

64 GB workstation — maximize throughput

terminal

export MEMTRACE_TIER=heavy
export MEMTRACE_EMBED_BATCH_SIZE=128
export MEMTRACE_EMBED_INTRA_OP_THREADS=8
export MEMTRACE_EMBED_QUANT=fp32
memtrace start

Skip embedding for structural-only passes

Parser and graph work dominate less than embedding on large repos. Skip semantic search entirely:

terminal

export MEMTRACE_SKIP_EMBED=1
memtrace index <path>

Or use a smaller model via memtrace embed set bge-small (~3× faster than jina-code, ~6 pts lower retrieval accuracy).

Faster queries — disable rerank

terminal

export MEMTRACE_RERANK=off
memtrace stop && memtrace start

Typical tradeoff: p50 ~50–150 ms instead of ~450–870 ms, with ~3–4 pp lower acc@1 on agent-style queries.

Repeated CI re-indexing

The on-disk embed cache at ~/.memtrace/embed-cache/ keys by symbol AST hash — unchanged symbols are cache hits even after wiping .memdb/. Mount the cache directory as a CI volume so embed passes become nearly free after the first run.

REMOTE EMBEDDERS SKIP LOCAL RAM PRESSURE

Remote embedding moves compute off your machine — the local RSS gate does not apply to HTTP embedding batches. Useful on constrained laptops and CI runners without GPUs.

Knobs and tradeoffs#

MEMTRACE_EMBED_BATCH_SIZE

Value	RAM	Throughput	Best for
4	~30% lower	~30% slower	RPi / very tight RAM
8	Light default	baseline	8 GB Mac
16	Standard default	baseline	16 GB Mac
64	Heavy default	best	32+ GB workstation
128	extreme	marginal gain	64+ GB GPU box

MEMTRACE_EMBED_INTRA_OP_THREADS

Doubling threads roughly doubles per-op scratch RAM. Above 4 on a non-GPU host rarely helps — embedding is usually memory-bandwidth-bound.

MEMTRACE_RERANK

Setting	acc@1 (agent queries)	p50 latency	Extra RAM
off	~70%	~50 ms	0
on (default)	~74%	~450 ms	~75 MB int8 model

MEMTRACE_EMBED_RSS_LIMIT_GB

Soft ceiling — crossing it yields the embed loop for 50 ms and logs a warning. Do not set so low that it fires constantly; defaults scale with host RAM.

MEMTRACE_UNIFIED_CACHE_MB

In-process hot cache (default 256 MB). Raise to 512 or 1024 on RAM-rich hosts; 0 disables.

MEMTRACE_ORT_LOW_RSS

Disables ORT memory arenas on every ONNX session site — empirical −2.7% RSS for −19% throughput on shipping model sizes. Off by default.

Diagnose without changing anything#

terminal

memtrace status
# data dir, graph counts, noise-filter skips on the running daemon

Daemon logs (foreground memtrace start or log files under ~/.memtrace/):

log lines

ort: global thread pool capped — intra_op=2, inter_op=1
embed: RSS sample batch_idx=32 rss_mb=4892 limit_mb=6144

The dashboard Value tab breaks down per-query latency. For embed provider health: memtrace embed status and memtrace embed test.

NOTE

Trim indexed noise with .memtraceignore before chasing performance — fewer files means faster walks and cheaper embed passes.

See also Troubleshooting for circuit-breaker and ONNX load failures, and memtrace warmup to pre-pay CoreML compile cost.