Sharding and scale

The memory bill arrives first

A float32 vector (four bytes per number) of 768 dimensions is about three kilobytes. Ten million of them are thirty gigabytes before any index overhead, and a graph index adds edges on top. The first scaling problem is usually RAM, not query throughput. So a machine that answers ten thousand queries a second can still be unable to hold the corpus.

That number is also the argument for quantisation (storing each number with fewer bits): int8 cuts it by four, product quantisation by more, at a cost in recall. Decide which cost you are paying before you add machines, because sharding is the most expensive way to buy memory. Quantisation is a setting change; a shard is a new failure domain, a new backup and a new thing to monitor.

The arithmetic is worth doing before the design diagram. A single node with 64 GB of RAM and a graph overhead of 1.5 holds roughly fourteen million 768-dimensional float32 vectors, with nothing left for the operating system. That is the real capacity, not the number on the instance page. It is the number that decides whether the next conversation is about quantisation or about shards.

Sharding by hash or by tenant

Hash sharding spreads vectors evenly and makes the query a scatter-gather (ask every shard, then merge). Every shard returns its top k, and a merger takes the global top k. It scales writes and reads evenly and makes cross-tenant queries impossible to avoid. The price is that every query pays for the slowest shard and every incident involves more than one machine.

Sharding by tenant puts each customer on one shard, so a query touches one machine and a delete is local. It also creates hot shards when one tenant is much larger than the rest, which is a capacity planning problem rather than a search one. The routing table becomes the most important piece of state in the system, and it needs a backup.

The index, inside Postgres

import psycopg

with psycopg.connect(dsn) as conn:
    conn.execute(
        "CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops) "
        "WITH (m = 16, ef_construction = 64)"
    )
    rows = conn.execute(
        "SELECT id, embedding <=> %s AS distance "
        "FROM items ORDER BY distance LIMIT 5",
        (query_vector,),
    ).fetchall()

Replication changes the answer

Replicas are how you survive a machine and how you serve more queries, and they introduce lag. A write acknowledged by the primary may not be visible on the replica that answers the next search. So a user can add a document and fail to find it. That is not a cache problem, and clearing the cache will not fix it.

The fix is a consistency promise, not a bigger cache. Route reads that must see their own writes to the primary, or carry a version and wait until the replica has caught up. Most search products can tolerate a second of lag; a delete usually cannot. Write the promise down as a number and test it with a clock, not with a feeling.

The cost model to write down

Every shard multiplies the fixed cost: memory, replicas, snapshots, and the operational attention a failure needs. The number that decides the shard count is the largest corpus one machine can hold with headroom, not the total divided by a round number. Headroom is not optional. Compaction, a rebuild and a traffic spike all want the same memory as the steady state.

Write the model down before the incident: vectors × dimensions × bytes, plus index overhead, plus a replica factor, plus headroom. A rough number that everyone agrees on is worth more than a dashboard nobody trusts. Recompute it when the embedding model changes, because a dimension change moves every term at once.

The model also gives you a test: double the corpus and the shard count should roughly double, unless quantisation or a shorter vector moved the per-vector cost. If the plan says otherwise, one of the terms is stale. The stale term is usually the dimension of the current embedding model. Recompute the plan when the model changes, not when the incident does.