The Architecture of AI-Native Query Pipelines

Solving the Latency Crisis Through Cost-Based Optimization and Multi-Stage Vector Execution

Written by: Arunkumar Mathiyazhagan, IEEE Senior Member & AWS Senior Software Engineer

Scope note: This is a practitioner synthesis article, not an empirical research paper. Latency figures are representative estimates from production-scale deployments and cited literature, not controlled measurements from a single system. Claims are scoped to the workload characteristics described. Readers building production systems should validate against their own corpus size, network topology, and hardware configuration.

Key Takeaways – In large-scale, networked RAG systems, data movement — not model inference — is the dominant latency source: 40–80ms vs. 10–20ms for typical high-throughput deployments. – A multi-stage query planner (metadata pruning → ANN → in-engine re-ranking → inference) can reduce payload size by orders of magnitude before the LLM ever sees a token. – The primary failure mode is planner misestimation: stale statistics cause the optimizer to pick the wrong execution path, often by 10x or more in skewed-distribution workloads. – Exqutor (arXiv:2512.09695, Dec 2024) shows that exact cardinality estimation in vector query planners cuts misestimation errors by up to 40% on filtered ANN workloads in their evaluated benchmarks. – Memory bandwidth — not SIMD throughput — is the true ceiling. Data layout (Apache Arrow, columnar formats) determines whether vectorization helps at all.

1. The 3 AM Problem: When the Bottleneck Isn’t Where You Think

It is 3 AM. Your RAG pipeline is spiking to 200ms p99 latency. You pull up the traces. Model inference: 18ms. Everything else: 160ms.

This is the scenario that separates engineers who have run AI systems in production from those who have only benchmarked them in notebooks. The industry has spent enormous energy optimizing transformer inference — quantization, speculative decoding, flash attention. Those gains are real. But in high-throughput retrieval systems, the bottleneck has quietly migrated upstream, into the data layer, where it is harder to see and harder to fix.

The culprit is what I call the Serialization Tax: the overhead of treating a vector store as a passive repository and moving raw, unfiltered data across the network for application-layer processing. In large-scale, networked RAG deployments — the systems where this bottleneck is most acute — a query against a 100-million-row corpus does not fail because the LLM is slow. It fails because the system ships 50,000 candidate vectors to the application tier, deserializes them, filters them, and only then hands a refined set to the model. Every byte that crosses that boundary is latency you paid for nothing. In smaller corpora or local inference setups, inference may well dominate; the Serialization Tax is a function of scale and network topology, not an absolute law.

Performance in these systems is bounded by how fast data can be fed to the CPU — a streaming workload — not by how fast instructions execute. That distinction changes everything about how you architect the pipeline.

2. The Multi-Stage Query Planner: Narrowing the Funnel

The architectural response to the Serialization Tax is an integrated multi-stage query planner that pushes computation as close to the data as possible. The goal is simple: at each stage, eliminate as many candidates as cheaply as possible before passing the remainder to the next, more expensive stage.

The Cold Cache Problem: Latency Before the Pipeline Starts

Before the first stage executes, there is a cost that most benchmarks ignore entirely: cache warm-up. Standard vector database benchmarks — ANN Benchmarks, VectorDBBench — measure query latency immediately after ingestion, when the HNSW graph is fully resident in DRAM and the OS page cache is warm. Production systems look nothing like this.

In a cold-start scenario — after a pod restart, a scale-out event, or a low-traffic overnight period — the first queries against an HNSW index pay the full cost of loading graph layers from disk into memory. For a 100M-vector index, this can add 200–500ms to the first query, and p99 latency remains elevated for minutes until the working set is warm. The “160ms everything else” from the 3 AM scenario is often not the steady-state cost — it is the warm-cache cost. Cold-cache cost is worse.

Practical mitigations: pre-warm indexes on pod startup by issuing synthetic queries against representative workloads before accepting live traffic; use memory-mapped files (mmap) to allow the OS to manage page eviction rather than loading the full graph eagerly; and monitor cache hit rates as a first-class operational metric alongside recall and latency. Teams that skip this step discover it the hard way when a deployment causes a latency spike that looks like a regression but is actually just a cold cache.

Stage 1 — High-Selectivity Metadata Pruning

Before touching a single vector, evaluate boolean predicates — temporal ranges, category filters, tenant IDs — directly on columnar formats (Apache Arrow, Parquet). These operations are cheap: bitmap ANDs, range scans on sorted columns. If your predicate has 3% selectivity against a 100M-row corpus, you have just eliminated 97M rows before spending a single cycle on vector math. This is the highest-leverage operation in the entire pipeline, and it is the one most commonly skipped in naive implementations.

Stage 2 — ANN Candidate Selection

With the filtered subset in hand, the planner invokes an approximate nearest neighbor index. Two dominant structures serve different workload profiles:

HNSW (Hierarchical Navigable Small World): A graph-based index optimized for low graph-traversal latency. Excellent recall at low ef values, but memory-intensive — the full graph must reside in DRAM. Degrades under high update rates as graph connectivity deteriorates.
Product Quantization (PQ/IVF-PQ): Compresses vectors into short codes, enabling compressed-scan efficiency at the cost of a small recall penalty. Memory footprint is dramatically lower — recent work on AiSAQ demonstrates ~10MB memory usage even at billion-scale datasets by offloading compressed vectors to storage.

The choice is not binary. Hybrid approaches (IVFPQ + HNSW) combine coarse quantization for fast candidate retrieval with graph refinement for precision, and are increasingly the production default for billion-scale workloads.

A note on Matryoshka Representation Learning (MRL). Both HNSW and PQ operate on vectors after they leave the embedding model. MRL attacks the problem one layer earlier — at the model itself. MRL-trained models (now shipping in OpenAI’s text-embedding-3 series, Voyage AI’s voyage-3-large, and Google’s Gemini Embedding 2) encode semantic information such that any truncated prefix of the full vector is itself a valid, lower-fidelity embedding. This means you can run ANN search on 128-dimension prefixes of a 1024-dimension vector for coarse candidate retrieval, then re-score with the full vector only for the top-k. SingleStore’s production benchmarks show this two-stage MRL approach reduces index memory by up to 87% and increases throughput up to 6.6x while preserving recall — gains measured on their specific workloads and hardware configuration, and likely to vary by embedding dimensionality and query distribution. That said, the directional result is consistent with the underlying mathematics of MRL: coarse-prefix search is strictly cheaper, and the recall penalty at the coarse stage is recoverable by full-vector re-scoring on the top-k.

Stage 3 — In-Engine Re-ranking

Rather than shipping ANN candidates to the application for re-scoring, a well-designed storage kernel performs initial distance scoring internally, returning only the top-k refined candidates. This is the stage where the payload shrinks from thousands of candidates to tens. Recent work on HyperRAG (arxiv:2504.02921, 2025) demonstrates that reusing document-side KV-cache during re-ranking cuts re-ranker inference cost by a significant margin while maintaining generation quality — a systems-level insight that makes in-engine re-ranking economically viable even for cross-encoder models.

The Bi-Encoder vs. Cross-Encoder Cost Decision. Not all re-rankers are equal, and the choice has direct latency implications that must be quantified, not assumed.

A bi-encoder scores query and document independently — embeddings are precomputed, and re-ranking is a dot-product operation. Latency: sub-millisecond per candidate. Weakness: the model cannot attend to query-document interactions, so it misses fine-grained relevance signals.

A cross-encoder concatenates query and document and runs them jointly through a transformer, allowing attention heads to evaluate token-level relevance. This produces materially better ranking quality — production teams report significant improvements in NDCG@10 and recall@k over bi-encoder-only pipelines, with gains varying widely by dataset, domain, and query distribution — but at a steep cost: one full forward pass per candidate pair. For a top-50 candidate set, a cross-encoder adds roughly 5x the latency of the ANN stage itself (based on typical BERT-base cross-encoder inference on CPU; GPU inference narrows this gap considerably).

The practical decision framework:

Scenario	Recommended Re-ranker
p99 latency budget < 50ms	Bi-encoder or MRL two-stage
High-precision retrieval, latency budget 100–200ms	Cross-encoder on top-20 candidates
Dynamic corpora, freshness-sensitive	Bi-encoder + periodic cross-encoder calibration

The emerging middle ground is late-interaction models (ColBERT-style), which precompute document token embeddings but perform query-document interaction at retrieval time. MICE (Vast et al., arXiv:2602.16299, February 2026) reduces cross-encoder inference latency fourfold while retaining most of its ranking quality — making cross-encoder-grade precision accessible at closer to bi-encoder cost. For teams currently skipping re-ranking entirely due to latency concerns, MICE removes the primary objection.

Stage 4 — Heavy Model Inference

The LLM receives a payload that has been pruned from 100M rows to perhaps 20 high-confidence candidates. At this point, inference latency is the dominant term — and it is operating on a fraction of the data a naive pipeline would have sent.

Figure: The four-stage query funnel. Each stage eliminates candidates before passing to the next, more expensive operation. A naive pipeline skips Stages 1–3 and ships the full corpus to the application tier — the Serialization Tax in action.

Quick Wins: Apply This to Your Pipeline Tomorrow

Not every team is ready to build a full cost-based planner. Here are four changes you can make immediately, in order of impact:

Add a metadata pre-filter before your vector search call. Even a simple WHERE tenant_id = ? evaluated on a B-tree index before ANN search can eliminate 90%+ of candidates. Most vector DB clients support this natively.
Switch to a columnar metadata store (Apache Arrow / Parquet). If your metadata lives in a row-oriented store, you are paying strided memory access costs on every filter. Columnar layout is a one-time migration with permanent latency benefits.
Pre-warm your HNSW index on pod startup. Issue 50–100 synthetic queries against representative workloads before accepting live traffic. Eliminates cold-cache latency spikes after deployments.
Add recall@10 to your monitoring dashboard. If recall drops without a model change, you have a graph degradation or over-filtering problem. You cannot fix what you cannot see.

3. Cost-Based Optimization: The Decision Framework (and Where It Breaks)

A sophisticated planner does not hard-code the execution order. It estimates the cost of each path and selects the cheapest one dynamically. Formally, for a query with predicate p and vector query q over corpus C:

Filter-First minimises total cost when sel(p) is small (empirically, below ~5% in most workloads): evaluate metadata first, pass the small surviving set C_filtered to the ANN index. The vector compute budget is spent only on records that passed the cheap filter.

ANN-First minimises cost when metadata is high-cardinality or poorly indexed — C_scan dominates — so the planner runs vector search first to narrow scope, then applies filters to the candidate set.

This mirrors classical cost-based query optimization in relational databases (Selinger et al., 1979; Graefe, 1993), where the optimizer selects join order based on estimated cardinalities. The key difference in vector systems is that C_ann depends on recall target as well as candidate set size — a coupling that relational optimizers do not face.

The Misestimation Problem — and the Research That’s Solving It

Here is the failure mode that senior engineers lose sleep over: planner misestimation. The optimizer’s cost model depends on statistics — predicate selectivity estimates, index cardinality, data distribution histograms. When those statistics are stale or skewed, the planner can choose ANN-first when filter-first would have been significantly faster, or vice versa. In traditional relational database literature, misestimation errors of 10x or more are well-documented on skewed distributions (Leis et al., “How Good Are Query Optimizers, Really?”, VLDB 2015); vector planners face the same problem with the added complexity of approximate recall semantics.

This is not a theoretical concern. In production systems with high write rates, statistics can drift within hours. A planner that was well-calibrated at index build time becomes increasingly unreliable as data evolves.

The most significant recent advance in this space is Exqutor (Park et al., arXiv:2512.09695, December 2024), a pluggable cardinality estimation framework for vector-augmented analytical queries. Rather than relying on approximate histograms, Exqutor leverages exact cardinality query techniques when vector indexes (HNSW, IVF) are available, feeding precise estimates back into the query optimizer. In benchmarks on filtered ANN workloads evaluated in the Exqutor paper, exact cardinality estimation reduces misestimation errors by up to 40%, directly translating to better plan selection and lower tail latency. This is the kind of result that changes architectural decisions: if your planner can trust its estimates, you can be far more aggressive about filter-first pruning without the risk of catastrophic misestimation on skewed distributions.

Complementing this, Quake (Mohoney et al., arXiv:2506.03437, June 2025) introduces adaptive indexing that adjusts partition structure to evolving access patterns, guided by a cost model that predicts query latency based on real-time partition sizes and access frequencies — achieving query latency reductions of 1.5–38x over static indexes on dynamic workloads in their evaluation.

4. The Memory Bandwidth Argument: Why SIMD Is a Secondary Optimization

There is a common misconception in performance engineering: that SIMD (AVX2/AVX-512) vectorization is the primary lever for fast vector search. It is not. SIMD is a secondary optimization. The primary lever is data layout.

SIMD instructions process multiple data elements in parallel using wide CPU registers. But they can only do this if data arrives at the CPU fast enough to keep those registers fed. If your vectors are stored in row-oriented formats, accessing a single dimension across millions of records requires strided memory reads that thrash the cache and stall the pipeline. The CPU is not compute-bound — it is memory-bandwidth-bound, waiting for data.

The fix is columnar storage. Apache Arrow’s columnar layout enables zero-copy memory access: a filter on a single metadata column reads a contiguous memory region, maximizing cache line utilization and enabling SIMD to operate at its theoretical throughput. Without this, you can have the fastest SIMD implementation in the world and still be bottlenecked by memory subsystem latency.

The practical implication: before profiling SIMD utilization, profile memory bandwidth utilization. If your CPU is spending cycles waiting for data rather than processing it, no amount of instruction-level optimization will help. Fix the data layout first.

The Hardware Horizon: CXL and Tiered Memory

The memory bandwidth ceiling is not static. Compute Express Link (CXL) is the emerging hardware answer to the Serialization Tax at the infrastructure level. CXL provides cache-coherent, low-latency access to memory attached via PCIe — effectively extending a server’s DRAM pool across devices without the serialization overhead of traditional network I/O.

CXL systems achieve 55–61% bandwidth improvement at balanced read-write ratios compared to flat DDR5 (per characterization in arXiv:2508.15980). CXL-enabled KV-cache deployments have demonstrated 21.9x throughput improvement and 60x lower energy per token in specific LLM inference configurations reported by Astera Labs at Supercomputing 2025 — results tied to their particular workload, baseline, and hardware; the directional improvement is consistent with CXL’s architectural properties but should not be taken as universally achievable. The CXL Consortium released CXL 4.0 in November 2025, doubling link bandwidth to 128 GT/s via PCIe 7.0 and introducing bundled ports capable of 1.5 TB/s connections. Microsoft has launched CXL-equipped Azure M-series VMs using Astera Labs’ Leo CXL Smart Memory Controllers, currently in private preview — available for customer evaluation but not yet generally available.

For vector search specifically, CXL enables a tiered memory architecture where hot HNSW graph layers reside in fast local DRAM while cold layers and PQ-compressed vectors live in CXL-attached memory, accessed with latency far below network round-trips. This directly addresses the HNSW memory wall — the constraint that forces teams to choose between graph quality and DRAM budget. As CXL becomes available in cloud environments, the architectural tradeoffs between HNSW and PQ will shift materially: the memory cost of maintaining large, high-quality graphs becomes manageable when the memory pool is no longer bounded by a single server’s DIMM slots.

5. Multi-Tenancy: The Security Failure Mode Nobody Talks About

The query planner’s metadata pruning stage — Stage 1 — is where tenant isolation either holds or catastrophically fails. This deserves more attention than it typically receives in performance-focused discussions.

In a shared vector index serving multiple tenants, a nearest-neighbor search is geometrically blind to ownership. The HNSW graph does not know which tenant owns which vector. If a tenant ID filter is applied after ANN search (post-filter), the graph traversal will happily surface candidate vectors from other tenants as intermediate neighbors during graph navigation — even if those candidates are ultimately filtered from the result set. The traversal path itself leaks information about the embedding space of other tenants’ data.

Research on multi-tenant vector databases (arxiv:2401.07119) identifies three isolation models with meaningfully different security profiles:

Silo (per-tenant index): Complete isolation, no cross-tenant graph traversal. High memory cost — index overhead multiplied by tenant count.
Pool (shared index with metadata filters): Memory-efficient but vulnerable to leaky neighbors if filters are applied post-ANN. Cross-tenant data leakage succeeds reliably against shared indexes without pre-filter enforcement.
Partition (tenant-scoped subgraphs): The emerging middle ground. HoneyBee (arxiv:2505.01538, 2025) formalizes this as a constrained optimization problem, dynamically balancing storage, query efficiency, and recall. Benchmarks show up to 6x faster query speeds than row-level security approaches with only 1.4x storage increase.

The security implication for query planner design is direct: tenant ID must be a Stage 1 predicate, not a post-filter. If the planner’s cost model ever routes a multi-tenant query through ANN-first execution without a tenant partition boundary, you have a data isolation failure regardless of what the application-layer filter does afterward. This is not a theoretical concern — it is the failure mode that turns a performance optimization decision into a compliance incident.

For teams operating under SOC 2, HIPAA, or GDPR, the planner’s execution path selection must be treated as a security control, not just a performance optimization.

6. When It Breaks: A Field Guide to Failure Modes

Recall Collapse Under Aggressive Filtering

Over-filtering is the most common production failure in hybrid search. When a metadata predicate is highly selective and applied before ANN search, the surviving candidate set may be too small for the ANN index to navigate effectively. HNSW in particular relies on graph connectivity — if the filtered subgraph is sparse or disconnected, the greedy search algorithm cannot find good neighbors, and recall collapses. The symptom is a sudden drop in retrieval quality that does not correlate with any model change.

Mitigation: monitor recall@k as a first-class metric alongside latency. When recall drops below threshold, the planner should fall back to a post-filter strategy (ANN-first, then filter) even at higher latency cost.

Planner Misestimation on Skewed Distributions

As described above, stale statistics cause the optimizer to choose suboptimal execution paths. The failure is insidious because it manifests as latency spikes rather than errors — the system returns correct results, just slowly. Skewed data distributions (e.g., a tenant with 10x the average document count) are particularly dangerous because they invalidate global statistics for local queries.

Mitigation: implement per-partition statistics refresh on high-write-rate indexes. Consider adopting frameworks like Exqutor for exact cardinality estimation on filtered workloads.

Index Freshness and Graph Degradation

HNSW graphs are built for a static dataset. Incremental updates — insertions and deletions — degrade graph quality over time as new nodes are connected with suboptimal edges and deleted nodes leave dangling references. In high-update environments, recall can degrade by 10–15% within days of the last full rebuild.

Mitigation: implement hybrid indexing strategies that maintain a small, fresh in-memory index for recent writes alongside the main HNSW graph, merging periodically. Monitor graph quality metrics (average degree, connectivity) as operational signals, not just recall.

7. A Concrete End-to-End Example

To make the pipeline tangible, walk through a single query against a 100M-document enterprise knowledge base.

Assumptions (stated explicitly): documents chunked to ~512 tokens, embedded to 1536 dimensions (OpenAI text-embedding-3-large), indexed with HNSW (M=16, ef_construction=200), metadata stored in Apache Arrow columnar format, inference on a GPU-backed endpoint (A10G), network-local deployment (no cross-region latency). Latency estimates are derived from published benchmarks for these configurations and should be treated as order-of-magnitude guidance, not precise measurements.

The query: “What is our refund policy for enterprise contracts?” from tenant acme-corp.

Stage 1 — Metadata pruning. Predicate: tenant_id = ‘acme-corp’ AND doc_type IN (‘policy’, ‘contract’). Assumed selectivity: ~2.8% (acme-corp holds 2.8M of 100M documents; policy/contract docs are ~40% of their corpus). Surviving rows: ~1.1M. Estimated cost: ~0.8ms (bitmap AND on columnar index, consistent with Apache Arrow filter benchmarks at this scale). Rows eliminated: 98.9M.

Stage 2 — ANN candidate selection. HNSW search over the 1.1M-row filtered subgraph, ef=128, returning top-500 by approximate cosine distance. Surviving candidates: 500. Estimated cost: ~8ms (consistent with published HNSW benchmarks at 1M-vector scale; full 100M-vector search would be ~40–60ms). Recall@500 remains stable because the filtered subgraph is large enough to maintain graph connectivity.

Stage 3 — In-engine re-ranking. Bi-encoder re-scores 500 candidates using full 1536-dimension vectors, returning top-20 by exact cosine similarity. Surviving candidates: 20. Estimated cost: ~4ms. A cross-encoder would improve NDCG@10 but add ~40ms on CPU (narrower gap on GPU); given the latency budget, bi-encoder re-ranking is the right call here.

Stage 4 — LLM inference. 20 candidates (~400 tokens each = ~8,000 tokens of context) passed to the LLM. Estimated cost: ~15ms on A10G (consistent with vLLM throughput benchmarks at this context length).

Estimated total: ~28ms. A naive pipeline — no metadata filter, ANN over 100M rows, full candidate set to LLM — would run ~180–250ms on equivalent hardware. The funnel reduced the candidate set by 99.998% before inference.

These are illustrative estimates, not measurements from a controlled experiment. The ordering of magnitude reductions at each stage is the architecturally significant result; exact numbers will vary with hardware, index parameters, query distribution, and data skew.

8. A New System Paradigm

Traditional databases optimize for disk I/O and CPU utilization. AI-native query systems must optimize for a different set of constraints:

Dimension	Traditional DB	AI-Native Pipeline
Primary bottleneck	Disk I/O	Memory bandwidth
Optimization target	Where data lives	How much data survives each stage
Index structure	B-tree, hash	HNSW, IVF-PQ, columnar bitmap
Cost model inputs	Row counts, index selectivity	Vector cardinality, recall-latency tradeoff
Failure mode	Lock contention, I/O saturation	Recall collapse, planner misestimation

The shift is conceptual as much as technical. Old systems ask: where is the data? AI-native systems ask: how aggressively can I eliminate data before it reaches the expensive stages?

This framing has roots in classical IR pipeline theory — cascade ranking architectures (Wang et al., 2011) and learning-to-rank systems (Liu, 2009) have long applied the same principle of cheap-first elimination. What is new is the combination of approximate vector search, cost-based plan selection, and LLM inference as the terminal stage — a coupling that requires rethinking the cost model from first principles rather than inheriting it from relational or BM25-based retrieval systems.

The research trajectory is clear. Exqutor’s exact cardinality estimation, Quake’s adaptive partitioning, HyperRAG’s KV-cache reuse in re-ranking — these are not incremental improvements to existing architectures. They are the building blocks of a query execution model designed from first principles for the retrieval-inference workload. The engineers who internalize this model now will be the ones debugging 3 AM latency spikes in five years — and actually knowing where to look.

The Full-Engine vs. General-Purpose Question

A peer reviewer posed a question worth answering directly: are teams moving toward purpose-built vector engines (Pinecone, Milvus, Weaviate) or toward adding planners on top of general-purpose systems (pgvector, Snowflake Cortex, DuckDB)?

The honest answer is: both, and the split is architectural, not preferential.

Full-engine systems win when the workload is vector-search-primary — when retrieval is the product, not a feature. They ship the multi-stage planner, the HNSW/PQ index management, and the recall monitoring as first-class primitives. The operational cost is a separate system to run, monitor, and pay for.

General-purpose systems with vector extensions win when the workload is analytics-primary — when you need joins, aggregations, and vector search in the same query, and the operational cost of a separate vector store outweighs the performance ceiling. pgvector’s halfvec support and Snowflake’s Cortex Search are closing the performance gap faster than most expected. The TCO question is real: a purpose-built engine like Pinecone or Milvus can deliver 5–10x lower query latency than pgvector at scale, but it also doubles operational surface area — a second system to provision, monitor, tune, and pay for. For most teams below 50M vectors, the general-purpose path wins on TCO. Above that threshold, the latency savings of a dedicated engine typically justify the complexity.

The emerging pattern at scale is a hybrid: a general-purpose OLAP engine (Snowflake, BigQuery, DuckDB) handles metadata filtering and analytics, while a purpose-built vector index handles ANN search — with a query planner that routes across both based on the cost model described in this article. This is not a compromise; it is the architecture that Exqutor’s pluggable cardinality estimation was designed to enable. The planner becomes the integration layer, and the choice of underlying engine becomes a deployment decision rather than an architectural one.

The Index Freshness vs. Query Optimization Trade-off

A data scientist reviewer posed the question directly: how do you handle the trade-off between index freshness and query optimization in practice?

This is the hardest operational problem in production vector systems, and it does not have a clean answer — only a set of explicit trade-offs that must be made consciously.

The tension is structural. Exqutor-style exact cardinality estimation is computed at index build time. Every write — insert, update, delete — potentially invalidates those statistics. The more aggressively you optimize the query planner, the more sensitive it becomes to stale statistics. High write rates and high query optimization are in direct conflict.

The practical resolution is a tiered freshness architecture:

Tier 1 — Immutable base index. The bulk of the corpus lives in a fully optimized, statistics-rich index that is rebuilt on a schedule (nightly, weekly) rather than updated incrementally. Query planner accuracy is high. Write latency to this tier is zero — writes go elsewhere.

Tier 2 — Mutable delta index. Recent writes land in a small, separate index (flat scan or lightweight HNSW) that is always fresh but carries no optimizer statistics. Queries fan out to both tiers and merge results. Ada-IVF (arxiv:2411.00970) formalizes this as adaptive incremental maintenance — identifying which index partitions have degraded under updates and selectively re-clustering them, rather than rebuilding the full index.

Tier 3 — Statistics refresh cadence. Rather than continuous statistics updates (which add write latency), refresh statistics on a schedule tied to write volume. At 10% data change, trigger a statistics refresh. At 30% change, trigger a partial index rebuild. The thresholds are workload-specific, but the principle is: treat statistics freshness as a first-class operational metric with an SLO, not as a background maintenance task.

The cost of this architecture is query fan-out complexity and merge logic. The benefit is that the base index — which handles 90%+ of query volume — operates with full optimizer accuracy, while the delta index handles recency without contaminating the planner’s statistics. Most production teams that have solved this problem have arrived at some version of this pattern, whether or not they named it explicitly.

About the Author

Arunkumar Mathiyazhagan is a senior software engineer and IEEE Senior Member specializing in cloud-native architectures, distributed systems, and high-throughput data pipelines. He holds AWS Senior Software Engineer designation and has extensive experience designing industrial-grade infrastructure for latency-sensitive AI systems.

References

Park, K. et al. (2024, December). Exqutor: Extended Query Optimizer for Vector-augmented Analytical Queries. arXiv:2512.09695. https://arxiv.org/abs/2512.09695
Aguerrebere, C. et al. (2025). Cost-Effective, Low Latency Vector Search with Azure Cosmos DB. arXiv:2505.05885. https://arxiv.org/abs/2505.05885
Liang, S. et al. (2025). HyperRAG: Enhancing Quality-Efficiency Tradeoffs in RAG with Reranker KV-Cache Reuse. arXiv:2504.02921. https://arxiv.org/abs/2504.02921
Mohoney, J. et al. (2025, June). Quake: Adaptive Indexing for Vector Search. arXiv:2506.03437. https://arxiv.org/abs/2506.03437 (Note: submitted June 2025; cited here for its cost-model and adaptive partitioning concepts, which are consistent with the architectural direction described in this article.)
Matsui, Y. et al. (2024). AiSAQ: All-in-Storage ANNS with Product Quantization for DRAM-free Information Retrieval. arXiv:2404.06004. https://arxiv.org/abs/2404.06004
Zhang, Y. et al. (2025). HoneyBee: Efficient Role-based Access Control for Vector Databases via Dynamic Partitioning. arXiv:2505.01538. https://arxiv.org/abs/2505.01538
Patel, D. et al. (2024). Efficient Indexing for Multi-Tenant Vector Databases. arXiv:2401.07119. https://arxiv.org/abs/2401.07119
CXL Consortium. (2025, November). CXL 4.0 Specification. https://www.computeexpresslink.org/
SingleStore. (2025). Vector Search with Matryoshka Embeddings. https://www.singlestore.com/blog/vector-search-with-matryoshka-embeddings/
Malkov, Y. A. & Yashunin, D. A. (2018). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI. https://doi.org/10.1109/TPAMI.2018.2889473
Selinger, P. G. et al. (1979). Access Path Selection in a Relational Database Management System. ACM SIGMOD. https://doi.org/10.1145/582095.582099
Graefe, G. (1993). Query Evaluation Techniques for Large Databases. ACM Computing Surveys 25(2). https://doi.org/10.1145/152610.152611
Leis, V. et al. (2015). How Good Are Query Optimizers, Really? VLDB 2015. https://doi.org/10.14778/2850583.2850594
Liu, T.-Y. (2009). Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval 3(3). https://doi.org/10.1561/1500000016
Wang, L. et al. (2011). A Cascade Ranking Model for Efficient Ranked Retrieval. ACM SIGIR. https://doi.org/10.1145/2009916.2009934
Vast, M. et al. (2026, February). MICE: Minimal Interaction Cross-Encoders for Efficient Re-ranking. arXiv:2602.16299. https://arxiv.org/abs/2602.16299

Aguerrebere, C. et al. (2024).Ada-IVF: Incremental IVF Index Maintenance for Streaming Vector Search. arXiv:2411.00970. https://arxiv.org/abs/2411.00970