[D] Memory consolidation in LLM agents (implementation notes)

I’ve been experimenting with memory systems for agentic workflows and wanted to share a few observations from implementation side.

Context windows are finite. Naive approaches where you dump everything into context hit limits fast. RAG helps with retrieval but doesn’t really solve the consolidation problem.

I tried a few approaches. Pure RAG works for factual lookup but is terrible for temporal reasoning. Sliding windows preserve recency but lose long-term structure. Hierarchical consolidation seems more promising so far.

The hierarchical setup is loosely inspired by biological memory. There is immediate memory (raw observations, high detail, short retention), working memory (active context, medium detail, session-scoped), and long-term memory (consolidated facts, lower detail, persistent).

One thing that surprised me is that consolidation isn’t just compression. It’s closer to abstraction. For example, turning “user fixed a bug in auth.py after adding verbose logging” into something like “user prefers explicit error handling in auth-related code”.

My current implementation uses sqlite for structured facts, a vector DB for semantic retrieval, and an LLM-based consolidation pipeline that runs asynchronously after conversations or task boundaries.

Retrieval strategy ended up mattering more than storage. Pure semantic search misses temporal structure, while purely temporal retrieval misses semantic connections. A hybrid approach with routing based on query type has worked best so far.

While looking at related work and evaluation efforts around agent memory, I stumbled across the Memory Genesis Competition. It seems like a lot of teams are independently converging on similar consolidation and retrieval problems.

Anyway, that’s roughly where things are at. Still iterating on the consolidation logic and trying to understand where it breaks down.