BlogLLM Infrastructure · Oct 20266 min read
P99 is a UX Metric. Uber Just Hid It in an SLO.
Uber's billion-scale vector search wasn't an AI story. It was a product team deciding what "reliable" means at 11pm in the rain.
The Setup
Uber published an engineering post about scaling vector search to a billion data points using OpenSearch. The technical execution is impressive. But the product decision buried inside it is more interesting than the architecture.
They set a target: P99 latency under 100ms at 2,000 queries per second.
Most PMs read that and move on. It sounds like an infrastructure constraint. It is not. It is a product decision about what reliability means for a specific user in a specific moment.
What P99 Actually Measures
P99 means 99% of users get search results in under 100ms. The other 1% wait longer.
At 2,000 queries per second, that 1% is 20 users every second. At Uber's scale, across millions of daily searches, that number compounds fast.
Now add context. A user searching for a cab at 11pm in the rain, running late for a flight, or trying to get home after a concert is not in a forgiving state. A 250ms result that stutters does not feel slow. It feels unreliable. One stalled booking in that moment changes how the product feels for that user permanently.
This is why Uber did not optimize for average latency. They optimized for the tail. Because the tail is where trust breaks.
The System Got Smarter. Then It Got Unpredictable.
Uber moved from keyword search to vector search. Instead of matching "cheap cab to airport" literally, the system now understands that "affordable ride to terminal" means the same thing. Genuinely useful upgrade.
But vector search is approximation. It uses a graph of similar data points where the nearest neighbor is the most relevant result. When you distribute that graph across machines, three things happen:
Every request splits across servers. The slowest server determines the final response time. Background indexing competes with live user traffic for memory.
This is latency variability. And it is a direct product of making the system more intelligent.
The engineers called it the Straggler Effect. Nine nodes returning results in 50ms and one node taking 180ms means your P99 is effectively 180ms. Not because the system is broken. Because distributed systems have noise.
The product implication: improving system intelligence increased complexity, and complexity increased unpredictability. Smarter search introduced a new category of failure that keyword search never had.
The Decisions That Were Actually Product Decisions
Uber made a series of tuning choices to get from P99 at 250ms down to P99 at 100ms. Each one looks like a configuration change. Each one was actually a tradeoff with a product consequence.
Replicas over cost. Adding more replica copies of each data shard meant queries could be routed to the fastest available copy instead of waiting for a slow one. This smoothed out the Straggler Effect. It also increased memory cost. Someone decided reliability was worth that cost. That is a product decision.
K value over recall. K is the number of nearest neighbors the system retrieves. Higher K means better search quality, more relevant results, fewer misses. It also increases latency. Uber tested the tradeoff and found latency increased only marginally with higher K. They accepted the cost. That is a product decision.
Blue/Green clusters over simplicity. Rather than reindex on the live cluster and risk latency spikes during updates, Uber ran two separate clusters. One serves traffic. One rebuilds the index in isolation. When the new index is ready, traffic switches instantly. This decoupled indexing from serving entirely. It added operational complexity. Someone decided stability under update was non-negotiable. That is a product decision.
What PMs Miss When They Read Infrastructure Posts
The question most PMs ask when reading something like this is: what did the engineers build?
The more useful question is: what did the product team decide to treat as non-negotiable?
Uber decided that tail latency, not average latency, was the reliability metric that mattered. That one decision cascaded into every architectural choice downstream. The replica strategy, the memory allocation, the Blue/Green deployment, the refresh rate tuning all of it follows from that single upstream product constraint.
P99 under 100ms was not handed to the engineering team as a technical requirement. Someone decided that a user booking a ride in a high-stakes moment deserved the same experience as a user browsing casually on a Tuesday afternoon. That is a product stance, written in infrastructure.
The Takeaway
The next time you read an engineering post about latency optimization, look for the number they chose to optimize for and ask why that number.
It will almost always trace back to a specific user, a specific moment, and a decision someone made about what that user deserved.
That decision is yours to make. Not the engineers'.