Teaching my app to remember what I meant last Tuesday

This happens to me about four times a day.

I find a bug. I explain it to my coding agent. We fix half of it. Three days later the other half turns up, and I'm explaining the whole thing again from scratch to an agent with the memory of a goldfish.

"No, not that price. The one on the pricing page. Remember? We tried the cache thing? It didn't work?"

It doesn't remember correctly.

So last week I gave Deiko a memory. Quick refresher if you're new here: you double-tap a key, point at things on your screen and talk, and Deiko turns that into a brief (your words, crops of what you pointed at, the exact text on screen) that you drop into Claude Code, Cursor or whatever agent you use. Every brief also lands on a board. That board is where the problem was hiding.

The problem with a pile

After a few weeks of use, my board was a pile. Eleven briefs from one design demo sat there as eleven loose cards. Three briefs about the same $99 price bug had, as far as Deiko knew, nothing to do with each other.

Each brief knew everything about its own moment and nothing about the one before it. What I wanted was easy to say:

Talk about the same bug on Tuesday and on Friday. Deiko knows it's the same bug, and Friday's agent knows what Tuesday's agent already tried.

Easy to say, anyway.

Enter Jev

Jev kept turning up on my X feed. It's a model from TypeSafe AI that does one narrow thing oddly well: you give it some facts and a typed question like "is this X?", and it gives you back a probability. You don't get an essay or a "Great question!", just a number.

That's exactly the shape of my problem. "Does this new brief belong to that old task?" is a yes-or-no question with a confidence attached. It's also cheap: TypeSafe's docs list $0.042 per million input tokens, and output is free. Sorting one brief costs about a tenth of a cent.

So I went to sign up, full of momentum, and got this:

Whoops, we're full - check x.com/typesafeai for more information!

They had paused signups two days after opening them. Cool. Great. Love that for me.

Luckily the same model is resold at the same price through a few gateways, so I wired in Vercel's AI Gateway, OpenRouter and Cloudflare, and the week carried on.

Homework: how does everyone else do this?

Before writing any code I did the thing I usually skip: I read how other tools remember things. I skipped the landing pages and read the mechanisms: the docs, the source where I could, and the GitHub issues where people complain.

One bug report stayed with me: a memory tool had saved the same memory 808 times, because it kept re-learning whatever it had just put into the prompt. Lesson noted: never learn back what you injected.

Three versions in one week

v1: chains. Each brief pointed at the one it continued. The result looked like beads on a string, and it barely helped. The eleven demo briefs were still eleven cards.

v2: tasks. Now a brief belongs to a task, and a task belongs to a project. Deiko keeps a note for each task: where it stands, what was decided, what was tried. When the agent finishes, it writes its outcome back into that note. To pick the task, I showed Jev a shortlist of 8 and asked "which one is this most like?"

I shipped it, re-sorted my real board and opened it up feeling rather proud.

The part where my own board embarrassed me

• A question about a signup chart ("so here in week 32 chart…") was filed under Pricing, at 86% confidence.
• A mic check, which was literally me saying "testing, testing, can you hear me", was filed as real work. Under Pricing too, naturally.
• Every brief lived in a project called "Deiko", including the ones that had nothing to do with Deiko.

When I dug in, every mistake was mine, not Jev's.

1. I asked a question that always has a winner. "Which of these 8 tasks is it most like?" is like asking "which of these 8 people is your cousin?" when none of them is. Someone gets picked anyway.
2. Nothing asked "is this even a request?" So "hello hello" got filed like a bug report.
3. My shortlist favoured recent work over related work. Pricing was always in the running because I'd touched it recently.
4. One wrong join made the next one worse. I described each task by its newest brief. Once the mic test joined Pricing, Jev saw Pricing described as "they are testing the microphone". It snowballed.
5. I was ignoring my best clue. Chrome tells Deiko the page name ("Signups — build", "Pricing — build"). I saved it and never used it.

I was giving Jev a multiple-choice quiz when it's at its best with yes or no.

v3: a very careful librarian

v3 files a brief the way a careful librarian files a note.

1. Is it a note at all? Jev answers "is this a real request?" Mic checks, greetings and thank-yous go to a pile called mic checks & scraps.

2. Read the labels. Before any AI gets involved, Deiko reads exact labels off what it already recorded: the page, the file, the project, the error, the ticket number. Each one is tidied so two visits match. "(3) Signups — build" becomes "Signups". "/users/8812" becomes "/users/*". An error loses its line number (thanks, Sentry). Titles that don't name a real page, like "New Tab" or "Dashboard", don't count.

3. Cast a wide net. Up to 20 tasks get a seat at the table:

• An exact label match gets a seat outright, unless that label is on more than 5 tasks. Looking at you, index.tsx.
• Time words like "yesterday" or "last week" get a few seats. So do "kal" and "pichle hafte", because I narrate half my briefs in Hinglish.
• The remaining seats go to a blend of keyword search (BM25) and meaning search. The meaning part is a small model that runs on your Mac (Microsoft's harrier-oss-v1-270m, about 200 MB, around 70 ms a brief), so "the graph thing" still finds your chart work even when no word matches.

Recency gets no seats any more. It only breaks ties.

4. Ask yes or no, once per task. In one request, Jev answers "is this the same piece of work?" for each of the 20 tasks. Every task gets its own probability, and they don't have to add up to 100%. The question says plainly that the same app or the same topic isn't enough.

5. Look twice at the finalists. The one or two tasks that scored 35% or more each get a private second request containing just the new brief and that one task, so nothing else can sway the answer. It asks the same question again, plus one more: are these different, related but separate, or the same work?

6. Decide in boring code. Jev only gives numbers. The rules that turn numbers into decisions live in my code, in one place, with tests:

export const GATE = 0.5;  // below this, it's a mic check
export const JOIN = { first: 0.6, second: 0.4, gap: 0.2, recent: 0.5, recentMs: 30 * 60e3 };
export const ASK = 0.35;  // between this and a join: ask the human

• A brief joins a task only when the second look says "same", round one is at 60% or more, and that task is 20 points ahead of the runner-up.
• Work from the last 30 minutes on the same page or file joins a little more easily. Old tasks never get that break, so they can't quietly swallow new work.
• A different ticket number (ENG-142 against ENG-150) never joins.
• For a maybe, the brief starts on its own and its card asks "Which one?" with the likely tasks as buttons. One click fixes it.
• Related but separate work gets a "Related to Pricing" link instead of a merge.

Plot twist: Jev is shy

My design doc said "join when the second look is 90% sure". Sensible and principled.

It joined nothing. Zero briefs.

Before building v3 I had made a test set from my own board: real briefs, each labelled by hand with where it belongs. Every change had to beat the previous run. On the ten briefs that truly belonged together, Jev's one-on-one "yes" sat between 0.43 and 0.88. It said "same" every single time. It just wouldn't say it loudly. Meanwhile, every brief that really was new had its best match at 0.23 or lower.

So the numbers in that snippet came from my data, not from a hunch. If you take one thing from this post, make it this: build the test set first. I was confidently and completely wrong about the thresholds.

Where it stands now: 28 of the 33 labelled briefs land where I'd put them by hand, and about one real brief in nine gets a "Which one?" card. I watch that second number closely. If it creeps past one in four, people start clicking anything to make the card go away, and then memory is worse than no memory.

Letting the agent dig for itself

Filing will sometimes be unsure, so the agent shouldn't have to depend on a single guess.

When a brief joins a task, the agent is told: this carries on from "Pricing display bug", where it stands, and the last few briefs, one line each. When it's only a hunch, the prompt says so: possibly related, not confirmed. The whole memory section stays under 20 lines.

There's also a small memory helper, an MCP server that runs only on your Mac, with three tools: search_briefs, get_task and get_brief. Say "like the chart thing from last week" and your agent can go and find it itself. One button in Settings sets it up for every agent Deiko finds on your Mac.

The search covers everything on your Mac, but it hands back much less, because the answers go to your agent's cloud model. You get your briefs, the agents' notes and the screenshots you kept. Screenshots you removed stay removed, audio never leaves, and anything that looks like a secret is scrubbed first. That last rule isn't hypothetical: our security audit once found a live API key in text Deiko had captured from the screen.

What it feels like: on Thursday you point at the pricing page and say "the price still shows $99 after I edit it". Your agent tries a cache fix. On Monday you point at the same page: "still broken." Deiko files Monday's brief under Pricing display bug, and Monday's agent gets the note saying the cache fix was tried and didn't work, so it starts from there instead of from zero. There's a whole page on how memory works if you want the tour.

Things I'd tell myself a week ago

• Ask yes or no, never "pick one". A pick-one question always has a winner.
• Use exact clues before fuzzy ones: page names, file names, ticket numbers.
• Describe things by what you're sure of, never by whatever came in last.
• Keep the decision in your own code, with the numbers in one place.
• Build the test set before the feature. Your thresholds are wrong.
• When you're unsure, ask the human once, with buttons.

If you've built memory for anything, I'd love to hear what broke for you, and which of my thresholds you think will age worst. I'm @Deiko_App on X.