Keeping an AI Agent's Context From Ballooning With Step Count — Cache Breakpoints, History Folding, and Streaming

Shellby’s AI Agent runs commands on your server one step at a time: emit a tool call, read the output, decide the next step, until it’s done. The loop is the plainest manual loop — every step resends “the full conversation history so far” to the model.

The cost of plainness shows up with step count: a 20-step ops task resends all 19 prior commands and outputs on step 20. Token cost and prompt-processing time balloon linearly with step count, time-to-first-token gets worse the deeper you go, and the meter runs faster. This post covers the three fixes that push it back down, and why they stack.

First, locate where the bloat lives

The manual loop’s request body every step is system + tools + accumulated message history. Of that:

• system and tools are identical every step — yet reprocessed by the model every step;
• in the message history, early tool outputs (a wall of journalctl, a screenful of ps aux) are resent verbatim on every later step, but the model no longer needs their full text — it only needs to know “what that step did and whether it worked.”

Each fix targets one point: caching for “stable prefix reprocessed repeatedly,” folding for “old outputs resent repeatedly,” streaming for “waiting on the whole response before the first character.”

Fix one: prompt caching breakpoints

Anthropic’s prompt caching lets you place cache_control: ephemeral breakpoints in the request; the stable prefix before a breakpoint is cached server-side, and the next request that matches that prefix byte-for-byte hits the cache, skips reprocessing, and drops both latency and cost.

Breakpoints go in three places: end of system, end of tools, and the latest history message. The first two are naturally stable prefixes; the third is the key design — the breakpoint rolls monotonically forward with the last message. On step N the breakpoint is on message N; on step N+1 it rolls to N+1, so the “messages 1 through N” prefix hits the cache on the next step, and only the newly added message needs reprocessing.

[ system      ]  ← cache breakpoint (stable)
[ tools       ]  ← cache breakpoint (stable)
[ msg 1..N-1  ]
[ msg N       ]  ← cache breakpoint (rolls with the latest message)
[ msg N+1     ]  ← added next step, only this needs reprocessing

A pragmatic edge: custom endpoints (non-official Anthropic) ignore this field, so cacheControl is a configurable switch — on by default, effective on the official path, and it degrades harmlessly when pointed elsewhere. The request-body assembly is extracted into a static encodeRequestBody so it’s unit-testable off the network.

Fix two: history folding, and it must freeze monotonically

Caching solves “stable prefix reprocessed,” but not “the history itself grows ever longer” — old tool outputs sit in the messages in full. The second fix is folding: keep the last N rounds (default 3) of tool results in full, and fold larger earlier outputs into a summary — just the exit code and first line, plus a “folded” marker. The model reads the summary to judge “did that step work,” and doesn’t need the screenful of raw output.

But folding has a trap that fights caching, and it has to be handled carefully: the act of folding mutates history, and mutating history breaks the cache prefix. If you re-decide “what to fold” every step, the prefix changes every step, the cache never hits, and the two fixes cancel out.

The solution is to make folding monotonically frozen: once an output is folded, it stays folded and its content never changes again; short outputs are never folded. That way folding causes a one-time cache miss only when an output first crosses the fold threshold, after which that prefix stabilizes and stops repeatedly breaking the cache. Folding only happens in the request body sent to the model; the live UI step stream still shows the full text, so the user sees the whole process.

Last 3 rounds:  [full stdout/stderr]
Earlier:        [exit 0 · "Reading package lists..." · ⋯folded 4.2KB]

Fix three: streaming, cutting time-to-first-token to the first token

The first two fixes attack “how much gets processed”; the third cuts “how long you wait to see anything.” The loop was a blocking single round-trip: if the model has a lot to say, you wait for the entire response to come back before it displays. Switched to SSE streaming, text deltas render as they arrive, and time-to-first-token drops from “wait for the whole response” to “wait for the first token.”

In implementation the provider is abstracted to stream() -> AsyncThrowingStream<AIStreamEvent>: it yields .textDelta to feed the UI preview and finishes with .completed (the full result — the tool-loop logic is untouched). The protocol ships a default implementation, so backends that don’t support streaming and the mock fall back to non-streaming with zero changes. Two real SSE providers each accumulate: Anthropic consumes content_block’s text_delta / input_json_delta, OpenAI accumulates delta content and tool_calls by index. Tool arguments are assembled incrementally in the stream and delivered whole at close.

An honest tradeoff: the streaming path does not reuse the “auto-retry on failure” logic — retrying after the first byte is semantically fraught (how do you count content you’ve already emitted?), so a failure before the first byte throws, and after the first byte there’s no retry. Reliability yields to the fact that you’re already streaming.

Why the three stack

The key is that they’re orthogonal: caching acts on “reprocessing the stable prefix,” folding acts on “history volume,” streaming acts on “waiting for the first byte” — three different axes that don’t interfere. Folding is deliberately designed to freeze monotonically precisely so it doesn’t break the cache prefix; streaming only changes “how the response is delivered,” never how the request body is assembled. So all three can be on at once: a long task both hits the cache, and doesn’t carry old outputs in full, and streams tokens live.

Takeaways

• A manual loop’s default behavior is linear bloat — untreated, more steps means slower and more expensive. First locate the two sources — stable prefix reprocessed, old outputs resent — then medicate each separately;
• Roll the cache breakpoint monotonically with the latest message, making the “history prefix” a reusable stable segment and letting only the newly added message bear the processing cost;
• Any optimization that mutates history must yield to caching discipline. History folding has to freeze monotonically — folded stays folded — or every step breaks the prefix, and the processing you save is less than the cache invalidation costs;
• Separate “how much gets processed” from “how long you wait.” Caching and folding cut the former, streaming cuts the latter; split optimizations by axis to confirm they’re orthogonal and stackable rather than fighting each other.