← Back to all essays2026-07-14 • 14 min read
#search-luceneSEARCH
Anatomy of a Segment Merge: Minimizing Lucene I/O Spikes under Sustained Ingestion
A benchmark measuring write amplifications across TieredMergePolicy vs LogByteSizeMergePolicy. Investigating page cache thrashing, flush stall anomalies, and zero-drop throttling strategies.
Anatomy of a Segment Merge: Minimizing Lucene I/O Spikes under Sustained Ingestion
In large-scale search architectures (whether powered by Apache Solr, Elasticsearch, or custom Lucene wrappers), ingestion throughput is rarely limited by CPU. Instead, segment merges—the periodic background compaction of immutable inverted indexes into larger segments—act as the prime source of 99th-percentile search query latency cliffs.
The Compaction Bottleneck
When a flush occurs, in-memory index buffers are committed to new disk segments. Once the segment count exceeds threshold limits, Lucene triggers a merge.