Auto-Compact Ate My Constraints: A Post-Mortem

The incident first.

A long task: I asked an agent to refactor a module chain, and my first message stated the constraint explicitly — “do not touch module X; it has downstream dependents.” The first hour went perfectly. Two-plus hours and a few hundred tool calls in, the agent modified X.

It wasn’t disobedience, and it didn’t miss the instruction. The trace tells a clean story: when auto-compact fired, the summarizer ate the constraint. In the compressed context, “don’t touch X” simply ceased to exist. The model faithfully executed everything it could still see — the problem is that it could no longer see the rule.

The lesson generalizes into a judgment worth writing down:

The demo-to-production gap is mostly not a model problem. It’s two harness problems: constraint persistence and independent verification.

In a demo, the task finishes in ten minutes, the context never overflows, every constraint stays in the window. In production, tasks run for hours across hundreds of turns, and context will be compressed — context is volatile, but task constraints must be durable. Until that contradiction is resolved, every long task is a lottery ticket.

Fix #1: Externalize the spec — never into the compression pool

Constraints cannot live only in chat history, because chat history is what gets compressed. My approach: externalize the task spec into a structured file with four fields:

[locus:  where to start — file/module paths]
[signal: what evidence shows the problem — error log / failing test]
[done:   what counts as complete — verifiable acceptance criteria]
[done-v: verification mode — auto | manual]

Two mechanisms matter: the spec file never enters the compression pool, and after every compact, a re-read of the spec is forced. However mangled the chat history gets, “don’t touch X” returns to the model’s eyes verbatim, every time.

Fix #2: Compression must discriminate — no one-pot summarization

Auto-compact’s default behavior is “summarize the old stuff uniformly.” But information in a context differs wildly in importance. The right move is category-specific treatment:

Information type	Compression policy
Task spec & acceptance criteria	Lossless — not one word dropped
Key decisions and their reasons	Keep structured (decision + one-line why)
Intermediate observations (tool output, logs)	Aggressive summarization; large objects become indexed references
Failed attempts	Conclusion only (“approach A fails because B”); drop the process

The failed-attempts policy is the counterintuitive one. Keep too much, and residual wrong paths contaminate later judgment — the model gets dragged by its own history. Drop everything, and it will walk the same dead end again. Conclusion-only is the equilibrium between the two harms.

Fix #3: A consistency self-check after every compact

The mechanism is trivial: after each compact, insert one step — “Does the current plan still satisfy the task spec? Check line by line.”

It costs a few hundred tokens, and it converts constraint drift from post-hoc discovery (a human staring at the damage) into mid-flight interception (the agent notices its plan and spec disagree). I think of it as the long task’s re-alignment ritual:

Humans drift in three-hour meetings too. That’s why we invented agendas and minutes. An externalized spec and a post-compact self-check are the agent’s agenda and minutes.

A side observation: context is a layered budget, not a bucket

After this crash I rebuilt my long-task context strategy. The core shift: replace “one bucket that keeps filling” with four layers, each with its own discipline:

1. Fixed prefix (system prompt + tool definitions): stable, cache-friendly, untouched;
2. Task spec: resident and lossless, never compressed;
3. Working memory (current plan, key decisions): structured summaries;
4. Observation stream (tool output): aggressively pruned — retrieval instead of residency. Files don’t live in the window; they live in a local index and get fetched on use.

One selection note along the way: for local retrieval in desktop scenarios, my testing found SQLite FTS5 + BM25 more stable, cheaper, and more explainable than a vector database — full-text hits explain themselves; vector recalls often can’t say why they surfaced. Not every scenario needs embeddings, and that judgment saved me an entire vector-infra maintenance burden.

Closing

Long-task reliability is not solved by bigger context windows — however large the window, compression eventually fires and something gets dropped. It’s solved by one plain design principle:

Separate what is volatile (the conversation) from what is durable (the spec), and let no mechanism tie the latter’s lifetime to the former’s.

---

This is the second post of the Into the Harness series. Related: “Model + Harness = Agent: The Gap Isn’t Where You Think.”