Memory and history
The model forgets everything between calls, so memory is something you build: a list you keep, a rule for what to drop, and a summary for what cannot fit.
Memory is a list you own
There is no session inside the model. “Memory” is a list of messages that your application stores — in a variable, a database, Redis — and resends with the next question. The model’s contribution is only that it reads whatever list it is given.
That means every memory bug is a data bug. If the assistant forgets a name it was told, either the message is not in the list, or it was dropped by a trimming rule, or it is buried far enough back that the model weighted it less than something nearer.
A user says “my order is A-1024” in the first turn and asks about it ten turns later. If the reply is “which order?”, there are only three possibilities: the message was never stored, the window dropped it, or it is present but far from the end and lost among newer turns. Checking the exact list you sent distinguishes those cases in seconds; guessing at the model’s memory does not.
A window is the simplest rule
Keep the system message, then the last N turns. It is predictable, cheap and easy to explain. Its failure mode is equally predictable: the fact the user gave you at the start of the conversation is the first thing to fall out of the window.
Drop turns in pairs. Half a turn — a question without its answer, or an answer without its question — reads as a non sequitur and can make the next reply worse than having no history at all.
Trimming to a budget
from langchain_core.messages import trim_messages
trimmer = trim_messages(
max_tokens=1200,
strategy="last",
token_counter=model,
include_system=True,
start_on="human",
)
trimmed = trimmer.invoke(conversation)
reply = model.invoke(trimmed + [HumanMessage(question)])A window that keeps the system message
Run it, then shrink the window to one turn. Notice that the system message survives while the early facts disappear — that is the trade the window makes.
Summaries are lossy on purpose
A summary keeps the gist and drops the wording, which is the point. The order number, the deadline and the decision should survive; the small talk and the repeated confirmations should not. Write the summary prompt with that distinction in mind: list the durable facts first, then one sentence of context.
A common mistake is summarising every turn. The cost rises, the summary churns, and a fact can be paraphrased away by the third revision. Summarise in batches when the window is about to overflow, store the result beside the history, and keep the durable facts in their own record where no summary can lose them.
When the window is not enough
Long conversations need compression rather than deletion. Summarise the turns that are about to leave the window into a short paragraph, and keep that paragraph as a system-level message. The detail is lost; the fact that the user’s order number is A-1024 survives.
Summarising costs a model call, so it should happen in batches — every ten turns, not every turn — and the summary should be stored next to the history rather than recomputed from scratch.
Store the user’s durable facts separately from the transcript: name, plan, order id, preferences. A small key-value record survives every trimming strategy and costs almost nothing to resend.
What to keep out of history
Do not put secrets, full documents or tool payloads into the transcript. They are resent on every later turn, which multiplies cost and widens what a prompt injection can reach. Keep a reference — a document id — and fetch the content when it is needed.
The same goes for errors. A stack trace pasted into the conversation becomes context the model will try to explain on every subsequent turn.
A reference is also easier to revoke. If a document is deleted or a permission changes, the next turn fetches nothing, while a copy pasted into the transcript stays there for the rest of the conversation. The same rule applies to tool output: store the id and the summary, not the whole payload. History is the most expensive place in the system to keep anything you might need to take back. When in doubt, ask whether the next turn needs the bytes or only the reference; the answer is usually the reference.
Compress what falls out
The summariser turns dropped turns into one line. Add a durable fact — an order number — and confirm it survives the summary even after the turns are gone.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
Where does conversation memory live?