Lisbon AI
Agent
Memory.
in a nutshell:the do's & don'ts
Vitalii Ratushnyi
Lead Research Engineer · Harmix.ai · multimodal agentic memory
NotesI'm Vitalii, I lead research at Harmix, multimodal agentic memory. One running example the whole way through: a session in instinct, my personal coding agent, where I ask it to build me an agent that negotiates a better deal on my bills, on its way to v1. Every slide is a do or a don't, and every do comes back to one job, stated on slide 4: the next call gets the context the task needs, and the agent finishes the task instead of re-deriving it.
agenda
00
Two kinds of memory
01
Context window and MEMORY.md
02
The 1M window, measured
03
Don't brute-force
04
In a nutshell: four levels of state
05
Do: the handoff skill
06
Don't: structure-less memory
07
Do: a structured memory bank
08
Do: init memory bank
09
Do: init memory bank
10
Do: memory bank
11
Do: compression
12
Do: retrieval
13
Do: continual harness
14
Do: classify and retrieve with Jev
15
Links & questions
NotesDon't read the list. Name the shape: two don'ts, nine do's, one paper (Prime Agent, August 2026) and the links to all of it. Then move. Rows are clickable if Q&A sends me back somewhere.
00 · classification
Two kinds of memory.
class 01
Customer-facing agents
class 02
Personal coding agents
runs as
Semi-deferred. You build it for someone else, and it runs later.
Immediate. You build it for yourself, so you are the judge of whether it works.
remembers
The person. Taste, history, the option the user selected.
The project’s context. Decisions, constraints, dead ends.
without memory
Recommends the option the user turned down last time.
Retries the fix the last session failed at.
NotesThirty seconds, no more. The "runs as" row is my impression, not a taxonomy: the customer-facing agent is semi-deferred, you build it for a user who runs it later without you; the coding agent you build for yourself, and you are the judge of whether it works. The point of the split: everything about the window holds for both: it fills, it degrades, it drifts. What changes is what you write down and who gets to read it. Where it lives is the part I say rather than show: the customer store sits on a server, one per user, never read by a human, and a fix ships as a migration; the project bank sits in files next to the code, edited by hand, and a fix is one line. My examples run on the building side: instinct, my personal coding agent, building a deal agent that negotiates my bills, on its way to v1. The customer-facing twin is a Harmix agent picking music for a client spot; it forgets its brief exactly the way a build session forgets its plan. If someone asks which one this talk is about: both, and the mechanics are the same; the contents aren't.
01 · context window and MEMORY.md
The context window is RAM. Memory is the disk (not quite).
Memory has one job: hand the next turn the context the task needs.
* Codex and Claude Code both write notes to themselves and reload them at every start (~/.codex/memories across all projects, off by default; MEMORY.md per repository, capped at 200 lines or 25 KB), and whatever loads sits in the window on every turn.
NotesThe thesis is on screen as the slide opens; say it once, slowly, because every do and don't after this is an instance of it. Then the framing that carries the talk: the window is RAM, memory is disk. The parenthetical is the hook: say the qualifier out loud, because the analogy is wrong three times and each one is a later slide. RAM reads the same at every address; the window's middle degrades, and the next slide measures it. Disk pages in by itself; nothing here does until you write the retrieval, slide 11. RAM is bought once; the window is billed every turn. Name MEMORY.md out loud, and name its cap. Claude writes it, not the harness: the harness only stores the folder per repository, loads the index at the start of every session, and warns when it grows past 200 lines or 25 KB, beyond which the rest is not loaded. It holds four kinds of note: who you are and how you like to work, corrections you gave, project decisions the code cannot show, and where to find things. So it is a small decision log, but a shallow one, and it is machine-local. It is not free either: whatever loads, up to 25 KB of it, goes out with every request, though prompt caching prices most of that as a cache read. Everything after this slide is the memory you have to design yourself.
02 · the 1M window · measured
A 1M window is fake. The working range is about 400k.
GDM-MRCRv2 · 8 needles · top 7 of 87 models, sorted by AUC at 1M · 22 Sep 2026
contextarena.ai ↗
NotesThe table is sorted by AUC at 1M: the area under the recall curve from 8k out to a million tokens, one number for the whole length instead of one point on it. Only the top row has a score at 1M itself: gemini-3.7-flash, at 63.5. The other six sell a million-token window and have no 1M score on the board; a dash means not measured, not zero. Read the 128k column against the AUC column: gpt-5.6-sol holds 92.4 at 128k and its AUC to 1M is 56.5, and claude-opus-5 goes from 91.3 to 45.7. MRCR buries eight near-identical items in a long conversation and asks for one of them back, eight needles that look like each other, which is why it is harder than needle-in-a-haystack. Off the slide, if anyone pushes: sorted by the 1M column alone, second place is 9.3, from grok-4.20, sold with a two-million-token window, and the board prices that run at US$1,183. The 400k in the headline is my own rule of thumb, from my own evals rather than this board; say so if asked.
03 · don't brute-force
Neither more tokens nor a smarter model is good at compressing by default.
…
…
…
NotesStep through both, twenty seconds each. Tokenmaxxing is the one that does not work: same model, more tokens, and recall stops improving at ~400k while the bill does not. Modelmaxxing is the one the room believes, and it does work, for a while. Sorry Fable, you're the best model I've run, and you still wake up on Tuesday with no idea what we decided on Monday. Intelligence per turn is not persistence across turns, and that is where the smarter model stops. If someone answers "that's what RAG is for": three slides on. The pivot: neither one survives a restart.
04 · in a nutshell · same weights, different harness
The same model scores three times higher when the harness keeps its state.
30.2% → 95.5%
ARC-AGI-3, Opus 5, the same weights. The ARC harness against Prime Agent; the human baseline is 95.4%.
Prime Agent · Prime Intellect and Princeton · Aug 2026
arXiv 2608.23552 ↗
NotesLet the number sit for a beat before explaining it. The ARC number is the authors' own figure for the whole harness, REPL plus subagents plus an autonomous budget, and they say their reruns of native harnesses fell below published scores. Quote it as "same weights, different harness", never as a memory benchmark. The next slide shows where that state lives.
04 · in a nutshell · four levels of state
Memory has one job: hand the next turn the context the task needs.
L3
Disk-backed statehistory · artifacts · memories · skills · prompts · subagent specs
L2
REPL and subagentscode · tools · retained values · session state
model-context boundary
L1
Context Windowsystem prompt · AGENTS.md · tool definitions · messages · tool results
L0
Model weightsprior knowledge
…
…
…
Prime Agent · Prime Intellect and Princeton · Figure 2 · Aug 2026
arXiv 2608.23552 ↗
NotesThis is Figure 2 of the Prime Agent paper (Prime Intellect and Princeton, August 2026), and it corrects slide 01: not two levels but four, and the model reads exactly one of them. Step L0 to L3. The line to land on L1: compaction does not delete, it demotes. The prefix becomes a summary and the raw events fall to L3, where code can still reach them; that is why the pointers on the handoff card exist. On L2, the paper's phrase is "programmatic information management": the long document sits in a Python variable, the model searches it with code, and only the hit gets serialised into the window. The punch line names what memory is: a compressed record of what happened, when, and what came of it. Slides 09 to 11 take those three apart as facts, timeline and graph. The sentence I want the room to keep: a model should fail because the task exceeds its capability, not because the harness dropped state. Every slide after this one is about not dropping state.
05 · Do: the handoff skill
The default is a compact you don’t control.
Define your own with the handoff skill.
SESSION A · 700k / 1M
SESSION B · fresh window · 4k / 400k · your cap
…
# handoff · instinct harness · session 3
goal "build me an agent that negotiates
me a better deal"
state email on dry run, 20 threads
phone, chat: stubs, untested
constraints max €45/mo, hard
never share my card, said twice
decided auto-accept out: took €52/mo in evals
one offer per turn, logged
gotcha "lost counteroffer" in evals was a
reply-to header, not the agent
next dry run green: go live · red: fix parser
pointers s3://…/offers.jsonl · idx/deal-agent
NotesThree steps on the clicker: watch A creep toward a million, where it compacts on its own, write the briefing, start fresh. Name /compact first and be fair to it: it rewrites the window in place and the session continues, which is the cheaper move, and a custom compact, one you have written the rules for, works well. What it cannot do is choose to stop. The line to say out loud: a handoff is a briefing for a colleague, and the colleague is you, one window later; a summary of the chat is the wrong shape for that. Write it while the model still remembers why, not after recall has already faded.
06 · don't · vectors without structure
Chunk, embed, rank. Similarity doesn't really do precision well.
…
Recall without semantics and versioning is not memory.
NotesThe default architecture: chunk the transcript, embed, top-k at query time. One query: a commit hash. Embeddings don't do exact, so two swapped characters still rank first, and a near miss on an identifier is a miss. Similarity is a ranking, not a memory.
07 · do · memory bank
A memory bank holds the structured context of a project.
File-first, one set per project. The model reads it back to recall the state of the work.
## brief
instinct · deal agent for my bills · ship v1
## constraints
max €45/mo · never share my card (stated 2×)
## open
email on dry run · 20 threads running
## decisions
portal scraping out · captcha on 3 of 5 sites
auto-accept out · took €52/mo in evals
## procedural
evals lost counteroffers on the Reply-To header
→ thread by Message-ID, not Reply-To
## pointers
raw session → s3://…/raw.jsonl
embeddings → idx/deal-agent/v1
…
…
The memory is human-readable, greppable and open to Jev.
A loop improves only when its second pass knows what the first one tried, so it learns from past runs.
Notes"Controlled middle state" is the phrase to lean on: not the raw transcript, not the model's hidden state, but a file in between that you decide the shape of. Three modes: a core of under 10k tokens that loads every session, the rest fetched on demand, and writes that land after the turn ships as plain-text diffs, reviewed like code and reverted when wrong. Loops and graphs get overused (not my favourite pattern), but when you do run one, this file is what makes the second pass different from the first: auto-accept stays rejected, the Reply-To bug stays diagnosed.
08 · do · init memory bank
You can add a memory bank to an existing project with one command.
$ /init-memory-bankgist.github.com/MisreadableMind/e0e0e4cf
The bank prefills itself by reading the repo.
NotesThe animation runs on its own when the slide comes up. The point: you do not need a new project to start a memory bank. The command reads the manifest, the git log, the CI config and the linter config, and each file on the left answers a question the bank would otherwise ask you. Source: the init-memory-bank command, linked on the last slide.
09 · do · init memory bank
The bank works as a compressed index of your repo.
It extends to structured data sources, such as a SQL database or a vector store, through the same index.
.memory-bank/18 files · 5 folders
index.mdcore · summary, now, read and write rules
CHANGELOG.mdevery write: trigger, edit, effect, scope
product-overview/README · user-stories · features-epics
steerings/rules (core) · README · development-conventions
tech-details/README · tech-stack · testing · infrastructure
tasks/README · active · backlog
memory/README · timeline · graph
one line in AGENTS.md
Read .memory-bank/index.md and
steerings/rules.md before answering.
steerings/rules.md before answering.
the core
index.md and steerings/rules.md load every session and stay under 10k tokens. The rest is read when a question needs it.
The core feature is structure: the agent knows which lines are hard rules, so it stays inside them.
NotesThe tree fills in on its own; let it finish before talking over it. The command writes 18 files across 5 folders, pre-filled where the repo already answers the question. Four folders are plain project docs, one per topic, each with a README, so the bank grows by adding files. The memory layer sits on top: rules with their source in steerings/rules.md, events in memory/timeline.md, "why" in memory/graph.md, and every write logged in CHANGELOG.md with its trigger, so a wrong lesson is one revert. Skills, subagent specs and handoffs get a folder the first time one is saved, not at init. Everything is plain markdown, so a wrong guess is one edit, not a migration.
10 · do · memory bank
The bank holds three classes of information.
facts
What is true now
- One lookup, no scan.
- Session two starts from the constraints instead of asking for them.
- No history: the €45 max survives, the argument behind it does not.
- Goes stale without notice when the world changes.
max €45/mo
card never shared
email on dry run
timeline
The same facts, in order
- Every rule has an event behind it the next session can check.
- Tells a live decision from a reversed one.
- Grows with every turn.
- Needs conflict rules: when two events disagree, one has to win.
t09 max = €45/mo
t14 auto-accept = rejected, took €52
t28 email = shortlisted
t33 email = on dry run
graph
Why it is this way
- Answers "why" across several hops.
- Links each rule to the person or event that set it.
- Needs multi-hop search.
- A new edge that contradicts an old one has to retire it.
me ─set→ max €45
auto-accept ─violates→ max €45
Reply-To ─breaks→ offer evals
Facts without a timeline are rules nobody can defend. A timeline without a graph cannot say why.
NotesTwenty seconds, and it sets up the next two slides. Three questions, not three databases: what holds now, what happened and in what order, and why. Name the failure for each: without facts you re-ask every session; without a timeline a constraint has no defence and a reversed decision reads as live, so the agent redoes work you already undid; without a graph nobody can trace a rule back to who set it and what it blocked. Most memory products ship the first one only, which is why they answer state well and everything else badly. Next slide: the same session compressed into each shape.
11 · do · compression
Compression takes the shape of the question it will answer.
…
…
…
Facts answer state, a timeline answers order, a graph answers why and a skill answers how, so keep more than one.
NotesSame session, three shapes. Facts: what's true now, one hop, but no history. Timeline: what happened in order; causality survives, lookup gets expensive. Graph: "why" across hops: auto-accept → €52/mo → the €45 max → me, who set it. Skills: the Continual Harness step, where past sessions turn into code and subagent specs the next session runs instead of rereading. The write policy picks the shape; the question you expect picks the policy.
12 · do · retrieval
Memory is rebuilt for the question, almost never retrieved as is.
facts state
- never share my card (me, ×2)
- max €45/mo
- shortlist: email, chat, phone
- threads: by Message-ID
timeline history
- t11 · card rule (2nd time)
- t14 · auto-accept rejected · €52/mo
- t17–19 · offers lost → Reply-To
- t33 · email → dry run
graph why
- auto-accept ─violates→ €45 max
- me ─set→ no-card
- Reply-To ─breaks→ evals
context packet · rebuilt for this query · …
…
NotesCallback to slide 01: this is the page-fault handler, and you are the one who writes it. The model never sees the store; it sees a small packet assembled per query, under a fixed budget, so latency is a constant you can put in an SLA. Second query: the graph contributes nothing, and that's fine: stores are menus, not mandates. Land the parallel: human memory works this way too. Reconstruction, not playback, including the failure modes.
13 · do · continual harness · versioned memory
Memory that writes itself needs a version.
# memories · history
v5 t14 auto-accept rejected cause: took €52/mo in evalsv6 t20 thread by Message-ID cause: evals lost counteroffersv7 t33 email on dry run cause: shortlist lockedv8 t41 email shipped cause: dry run passed, none over €45
three rules for every write
Each write is a new version that logs the event behind it. Writes land between turns, never mid-turn. The system prompt is never edited, so rolling back to v7 is one command.
NotesThis is the write protocol from the Continual Harness half of the Prime Agent paper, and it is what slide 07's "written after the turn" means in practice. Each memory edit is a new version with its trigger logged, applied at the turn boundary. The base prompt stays immutable: refinement adds notes next to it and never rewrites it. So when a lesson turns out wrong, you roll back one version instead of starting the session over.
14 · do · classify and retrieve with Jev
Jev answers typed questions about the bank in milliseconds.
# jev · one request · four questions
state the bank + turn 41, verbatim
kind "never share my card"
fact | event | rule | procedure → rule · 0.94
stale "email on dry run", after turn 41? → yes · 0.97
relevant line 7, for "which channel do we ship?" → 0.88
leak card number in the outgoing draft? → no · 0.99
Launch figures: 70 to 500 ms per request, US$0.04 per million input tokens.
classification · on write
What kind of line is this?
Each new line gets a type and a probability before it lands. A low score goes to a person instead of the bank.
retrieval · on read
Which lines does this query need?
Jev scores every line against the question and keeps the top ones as written, so a hash like a3f9c21 survives intact.
An LLM rewrites what it retrieves. Jev only picks lines, so what it keeps stays verbatim.
NotesJev is TypeSafe AI's first "System One" model, out since 15 September 2026. It does not chat: you hand it a state and a set of typed questions, and it returns choices, scores and yes/no probabilities, several questions in one request. That shape fits a memory bank twice. On write, it classifies: is this new line a fact, an event, a rule or a procedure, which is the typing the next slide asks for, and the probability tells you when to ask a person instead. On read, it retrieves: score each line of the bank against the query and keep the top ones word for word, which fixes the slide 06 failure, because nothing gets paraphrased and an identifier comes back exact. The same idea runs the fast-jev-compaction plugin for Claude Code: it deletes tool calls Jev marks as stale instead of summarising, and a session near 1M came down to 86k in about a second. The latency and price are TypeSafe's launch numbers and the 86k run is a public demo, not my evals; say so if asked. The four answers on the card are illustrative.
15 · links & questions
Links, then questions.
init commandgist.github.com/MisreadableMind/e0e0e4cf
linkedinlinkedin.com/in/v-ratyshnyi
benchmarkcontextarena.ai · mrcr 8-needle
harmix · pammanager.harmix.ai/blog/spent-3000-on-memories
Vitalii Ratushnyi
Lead Research Engineer · Harmix.ai
Scan for the init command gist
NotesThis slide stays up for the whole Q&A; the links have to be readable from the back of the room. Q&A landmines: (1) "which vector DB": doesn't matter, the write policy does. (2) "won't 10M context solve this": the Context Arena board says 7.4% for the model that declares 10M; and a 10k-track catalogue is 67M tokens. (3) "isn't the handoff just compaction": compaction is the model guessing what mattered; the handoff is you deciding. That difference is the talk.