A vector store

Add, search, filter

Every vector database, hosted or embedded, offers the same three operations. add stores a vector with its text and metadata. search takes a query vector and returns the k nearest entries. filter restricts the candidates by metadata before or during the search.

The differences are operational: how the index is built, how it scales, how it persists, how it handles updates. The interface is small because the problem is small — everything else is engineering around speed and durability.

That is why swapping one store for another is usually a week of work rather than a rewrite. The parts that change are the client call, the index configuration and the way filters are expressed. The parts that stay are your chunks, your embeddings and the evaluation set that tells you whether the swap helped.

Exact search is a loop

Brute force is honest: compute the similarity to every stored vector and take the best k. It is exact, and it is fine for thousands of chunks — a few milliseconds in memory. Most small applications never need anything else.

Approximate indexes trade a little recall for a lot of speed at millions of vectors. The trade is invisible until it is not: an ANN index can miss the correct passage, and the answer is then wrong for a reason no prompt can fix.

The arithmetic explains the threshold. Five thousand chunks of 1,536 dimensions is about 7.7 million multiplications per query — trivial. Ten million chunks is fifteen billion, which is seconds per query. An index such as HNSW visits only part of a neighbour graph, and its search parameter decides how much to explore: raise it and recall rises with latency.

The same three operations, hosted

from langchain_chroma import Chroma

store = Chroma(
    collection_name="docs",
    embedding_function=embeddings,
    persist_directory="./chroma",
)

store.add_documents(chunks)

results = store.similarity_search(
    "How do refunds work?",
    k=4,
    filter={"product": "billing"},
)

Filter, then rank

A metadata filter is not a post-processing step. If you search for ten results and then discard the eight that belong to another tenant, you have eight fewer results than you asked for — and possibly none of the right one, because it was ranked eleventh.

Production stores apply the filter during the search so that the k results are the k best among the allowed candidates. When you write your own, filter the list before sorting, not after.

A worked case makes it concrete. Ten candidates, six of them for another product, and the one passage you need sits ninth. Ask for five and filter afterwards: it never appears. Filter first and it is in the allowed set, so it can be returned. Highly selective filters also interact with approximate indexes: some stores fall back to a full scan, which is correct and slower.

One collection per tenant

Separate collections are cheaper to reason about than filters and impossible to leak by forgetting a clause. They also make deletion honest: removing a customer means dropping a collection, not hunting rows.

Within one tenant, filters are still useful for narrowing by product, language or date. Those are relevance decisions, not security ones, and they belong in the query.

The operational cost is real: more collections mean more index builds and more configuration to keep in step. Start with one per tenant when the tenants are few and the data is sensitive, and with one collection and a tenant key when they are many. Either way, the boundary is a decision to write down rather than a clause to remember.

The store is an index, not the truth

A vector store can be rebuilt from the source documents, the chunker and the embedding model. That is its defining property: it holds a derived view, not the record. The moment it becomes the only copy of something, backups and migrations stop being an index problem and become a data-loss problem.

Treat it that way and the operational questions get easier. Re-embedding is a rebuild into a new collection, followed by a switch. A bad chunking change is a rebuild, not a repair. Deleting a document means deleting its chunks, and the source system remains the place where truth is decided.