Reducing the overhead of agent memory

Persistent memory adds work to a coding agent's loop. Records have to be selected, put into context, and updated when a decision changes. Each step can add a model request or repeat information the agent already has. This fork of Jay Zeng's pi-memory redesigns that loop for Pi, with learning handled by the working agent and recall built around complete, scoped records.

A topic index instead of automatic content injection

The original pi-memory extension (version 0.4.2) puts scratchpad items, daily logs, and a size-capped section of long-term memory into the system prompt. That makes recent material immediately available, but the character budgets decide which parts survive. A relevant decision can sit outside the retained portion, while unrelated daily activity still occupies context.

The fork separates discovery from reading. Each request receives the complete topic inventory for global memory and the focused project, plus explicit standing preferences. Other record bodies and scratchpad contents stay out until requested. Prompt content therefore follows the topic catalog and standing preferences rather than the full text of every stored record.

This also changes how a memory reaches the model. A clipped excerpt might contain ‘use a disposable database’ while leaving out ‘for integration tests in this project.’ Retrieval returns a complete record with its conditions. Larger records continue across whole paragraphs or code fences, so the agent can tell when it has more to read.

Explicit recall depends on the agent recognizing that a past decision matters. The topic inventory supplies discovery cues, and global preferences remain distinct from project-specific decisions. Task focus selects the repository being worked on, which can differ from the checkout where the session started.

Learn in the working session

The original extension can generate a model-written summary on quit. The fork removes that lifecycle inference pass. The working agent saves a settled decision while the relevant conversation is available, carrying its conditions and rationale into the record.

Source checking happens inside the write tool. A save supplies the current-session source reference and an exact quote, and the extension checks them against Pi's session tree. If the agent already understands the exchange, it can save directly instead of making another tool call to retrieve the same evidence. When context is missing, an evidence tool can read the necessary earlier exchange.

That separation keeps provenance without requiring the model to reconstruct the conversation for every write. Optional evidence views also omit earlier successful inspection payloads, preventing one inspection from expanding into another copy of a previous inspection. The source remains available through the session rather than becoming a transcript the model repeatedly consumes.

Return a usable memory from search

An earlier version of the fork returned excerpts centered on matching lines. The agent could find a relevant hit and still need a separate read to recover the complete decision. The current search response includes the full record when there is one strongest lexical candidate and the record fits the available context.

The fast path saves a tool round trip without forcing an arbitrary choice between equally strong matches. Ties remain short descriptions for the agent to select from, and other matches stay accessible. Once the agent has the complete applicable record, it can stop retrieving.

Search uses local MiniSearch ranking with identifier matching, exact phrases, prefixes, and typo tolerance. The query describes the decision the agent needs, rather than copying the whole conversation into retrieval. This keeps search independent of an embedding service or ranking model, at the cost of weaker matching when the query paraphrases a record without sharing its vocabulary. Topic browsing is the fallback.

Reuse the index without serving stale records

The earlier implementation rebuilt its MiniSearch index on every query. The current version reuses derived indexes keyed by the complete source contents. Each lookup still reads and validates the current ledger. A change to the indexed content produces a different key, so updates become visible without relying on a file timestamp to invalidate cached search state.

A local benchmark on an Apple M5 with Bun 1.3.13 used 1,024 synthetic records across 32 topics. It measured 80 repeated identifier searches, including ledger reading and validation, with a p95 lookup time of 13.8 ms. The topic index stays small because record bodies are retrieved separately. These measurements cover local retrieval, before any model or tool round trip.

• Memory extension and design notes↗
• Tool implementation↗
• Benchmark implementation↗
• Original pi-memory extensionJay Zeng's original project↗