Context Windows Aren't Memory: Why Your AI Assistant Forgets What You Just Told It

You correct the model once. It fixes the mistake, apologizes, moves on. Five turns later, in the same session, it makes the exact same mistake again - the one you already explained. It's tempting to conclude the model is being careless, or that you need to be sharper with your wording. Neither is really the problem. The problem is that you're picturing something the model doesn't have: memory.

What a context window actually is

When people say a model has a "128k context window" or a "1 million token context window," it sounds like a storage capacity - like RAM, or a database that holds your conversation and lets the model query it as needed. That's not what's happening. Every single time you send a message, the entire conversation up to that point - every turn, every tool output, every file the assistant read - gets re-sent to the model as one long block of text. The model doesn't "recall" turn three when you're on turn eight. It re-reads turn three, along with everything else, as part of a single pass over the whole transcript, every time.

This matters because it means there's no persistent internal state carrying your correction forward. The model isn't updating some internal belief that says "the user prefers snake_case" and consulting that belief later. If the instruction is still sitting in the text it's re-fed, it'll follow it. If that instruction has scrolled far enough back, or gotten paraphrased away in a summary, it's just gone from the model's point of view - not forgotten in the human sense, but literally absent from the input it's working with this time.

Why position in the context matters

Even when your instruction is still technically present in the context, where it sits changes how reliably the model uses it. This has been studied under the name "lost in the middle": models tend to weight information at the very beginning and the very end of a long input more heavily than information buried in the middle. It's not a hard cutoff - the model isn't literally blind to the middle of the text - but attention to it is measurably weaker, and errors climb accordingly.

Think about what a 40-turn coding session looks like from this angle. Your project's coding conventions, stated once at the top, are competing with a large volume of subsequent tool output, file contents, and back-and-forth for the model's attention. The convention hasn't been deleted. It's just increasingly diluted, sitting further from either end of a fast-growing pile of tokens the model has to weigh all at once. That's why the classic fix, "just repeat yourself," actually works: restating an instruction puts it back near the end, the position getting the most attention right now.

What happens when you blow past the limit

Every model has a hard ceiling on how many tokens of context it will accept. What happens when a conversation grows past that ceiling depends entirely on the tool wrapping the model, and the differences matter more than most people realize. Some tools will simply refuse the request with a hard error once you exceed the limit. Others quietly truncate: they drop the oldest turns and send only what fits, with no particular signal to you that anything was cut. Still others summarize: an earlier pass condenses old turns into a shorter recap, which then gets fed in place of the original text - lossy by construction, since specific details and exact phrasing can get flattened into a paraphrase that misses the part that mattered.

This is also why "just use a model with a bigger context window" is a partial fix at best. A bigger window delays the point where you run into truncation or summarization, but it doesn't eliminate the lost-in-the-middle effect - if anything, a much longer context gives the model more middle to lose things in. And most APIs charge per token, so a huge context you don't actually need is a real, ongoing expense, not a free upgrade.

Practical techniques that actually help

None of this means long AI-assisted sessions are hopeless - it means context has to be managed like any resource that gets stale and crowded over time. The single most effective habit is re-stating key constraints periodically rather than assuming they're still "in there." If you told the model an hour ago not to touch a particular file, and the session is still running, say it again before the next risky change.

A second technique that scales well for agentic coding sessions: keep a running decisions log as an actual file in the repo, not just as prior conversation turns. Have the assistant read that file at the start of a task and append to it as decisions get made. This turns your "memory" into something durable and external to the context window, rather than a fact you're hoping survived truncation.

Third, chunk large tasks instead of running one sprawling session. If you're migrating forty files, resist the urge to do it in one continuous conversation. Break it into smaller sessions per module, each with a fresh, short context that states exactly what's needed for that chunk. For agentic tools that support it, sub-tasks or sub-agents with their own scoped context serve the same purpose, letting the bulk of a large job happen outside your main conversation's growing pile of tokens.

The takeaway

Stop picturing the context window as a notebook the model is quietly filling up and consulting as needed. It's closer to a whiteboard that gets erased and rewritten in full every time you speak, and the model can only act on whatever happens to be legible on it that particular moment. Once you treat it that way, the fixes stop feeling like workarounds for a broken tool and start feeling like ordinary hygiene: restate what matters, externalize what needs to last, and don't ask one conversation to carry more than it can hold.