Tables and multimodal documents

A row without its header is a riddle

A table is a grid of cells whose meaning comes from the row and the column it sits in. Flatten it into a line of prose and the coordinates disappear: the text says “Q3 1200 980” and nothing says which figure is revenue and which is cost. The embedding then matches the topic and loses the relationship, so a question about cost retrieves the row and the answer quotes the revenue.

The fix is structural. Repeat the header with every row — “quarter: Q3; revenue: 1200; cost: 980” — or store the table as records and render each record as a sentence before embedding. A chunk that carries its own labels can be found, quoted and checked without the rest of the table beside it, and a citation can name the column it used.

Metadata matters as much as the text. A record chunk should carry the table id, the row number and the page, so a citation can point at the exact row rather than at a forty-page statement. That is also what lets a filter narrow a search to one table before ranking begins.

Numbers need a query, not a neighbour

“What was the total revenue in Q3?” is not a similarity question. No single row contains the total, and the nearest chunk to the question is likely the Q3 heading or a row that mentions revenue. Embeddings compare meaning; they do not add, subtract or group. Ask for an average and the top result will be a sentence about averages, not the average.

Route those questions to something that can compute. Keep the table as data beside the index — SQLite, DuckDB, a dataframe — and let the model call a query tool when the question asks for a sum, a count, a comparison or a trend. Retrieval supplies the schema and the column names; the tool supplies the arithmetic, and its result can be checked against the rows.

Extracting a table with its header

import pdfplumber

with pdfplumber.open("statement.pdf") as pdf:
    for page in pdf.pages:
        for table in page.extract_tables():
            header = table[0]
            for row in table[1:]:
                record = dict(zip(header, row))
                print(record)   # {"quarter": "Q3", "revenue": "1200"}

A scan is an image before it is text

A scanned page, a slide deck and a screenshot have no text layer. Something has to read the pixels: OCR for type, a vision model for a chart or a diagram. What you embed is that description, not the image, and the description is a summary written by a model that may have missed the axis label, the unit or the footnote that changes the meaning.

Keep the pointer. Every chunk produced this way should carry the page and the bounding box, so a citation can show the crop and a reader can check the original. The description is the search key; the image is the evidence. When the two disagree, the image wins, and a pipeline that cannot show it has no way to settle the argument.

Chunk figures with their captions, and never across a figure boundary. A caption that says “Figure 4: revenue by quarter” is often the only text that makes a chart findable, and splitting it from the image leaves both halves useless. The caption belongs to the figure the way a header belongs to a column.

Reading order is part of the extraction

A two-column PDF extracted naively interleaves the columns line by line, producing sentences that begin on the left and end on the right. Every word is present and the meaning is gone, and no chunker can repair it because the damage happened before chunking. The same failure appears in slides with two text boxes and in tables read row by row across merged cells.

Layout-aware extraction groups blocks by column before sorting them by vertical position, and keeps a caption with the figure it describes. That is the difference between a page that reads and a bag of text that happens to contain the same words. Reading order is not cosmetic: it decides which sentences exist, and therefore which chunks can be retrieved.