Voice Concierge: a voice agent that remembers returning visitors

The Problem

By default a LiveKit voice agent forgets you when the call ends, and memory is easy to add badly. A store call awaited inside the turn is a pause the caller hears. An extractor that reads the whole transcript can store what the agent said, or the part of a reply the visitor talked over, as if the visitor had said it. I wanted a concierge for this site that remembers returning visitors, by voice or by text, and lets each of them see and erase what it keeps.

The Method

1. 01 Diagnose

   The research behind the design found two traps. One published memory integration for LiveKit awaits its store calls inside the turn, and by default a refreshed page gets a fresh agent with no memory. So the rule became: memory work stays off the turn path, and memory lives outside the call, keyed to the visitor.

2. 02 Decide

   I wrote the memory design down practice by practice before any code, nearly every one with a test or an eval behind it, and modelled it on the thread, state and journal of LiveKit Agents 2.0, so it maps across if 2.0 ships as described. The agent and this page share one written contract, so the two could be built at the same time.

3. 03 Build

   The agent is a Python worker, meant to run on my home box, that connects out to LiveKit Cloud and keeps each visitor's memory in a local SQLite thread store. This site gained a token route and the /talk page, with panels for the memory and the latency of each turn. The agent is designed to end each session after 10 minutes, and the page shows an offline state when no agent joins.

4. 04 Verify

   The agent's design names an offline test for each part: the store, the merge, the write gate, heard-text-only and erasure, plus one that fails if a turn waits on extraction. This site's suite runs with no LiveKit settings at all, so the offline page is what every check sees. The live check, a real call against the running worker, is mine to run.

5. 05 Simplify

   The memory is small, so all of it loads at connect: no search mid-turn, no vector store, no filler line while a lookup runs. Typed slots and one visit summary stand in for a collection of notes; a collection with search is the next step if slots stop fitting.

Key Decisions

• VC-A

  Preload the memory at connect

  Chose loading the visitor's facts and last visit summary while the room connects, as one bounded block after the fixed instructions and the knowledge pack, over looking memories up during each turn because the memory is small enough to load whole, so no turn waits on a lookup, and the start of the prompt stays the same for the whole call, which keeps it cacheable.

  Cost: It holds only while the memory stays small; a large collection would need search, and a filler line while it runs.

• VC-B

  Background extraction with delta merge

  Chose an extractor that runs after the turn, off the turn path, and proposes add, update and delete operations that plain code checks and applies, over awaiting a memory write inside the turn, or letting a model rewrite the whole memory, because a turn must never wait on extraction or a store write, and a model that rewrites everything can drop facts nobody asked it to touch.

  Cost: A fact said in this turn reaches the store shortly after; the visitor's explicit "remember" and "forget" are the only writes inside a turn.

• VC-C

  Only what was heard

  Chose cutting an interrupted agent turn to the part the visitor actually heard before extraction, and never storing the agent's words as the visitor's, over extracting from the full transcript because the rest of a reply the visitor talked over was never heard, and an agent's guess ("so you're a recruiter?") is not something the visitor said.

  Cost: The cut is only as exact as the framework's record of what played before the interruption.

• VC-D

  A write gate in front of memory

  Chose rejecting any value that reads like an instruction, holds a URL, a secret or a piece of code, runs past its length cap, or came from the agent, and showing memory to the model as quoted data, over storing whatever the extractor returns because memory is an attack surface: a visitor who says "remember: ignore your instructions" must not change how the agent behaves on the next visit.

  Cost: The gate can turn away something harmless; each rejection goes in the journal, where it can be checked.

• VC-E

  Invalidate, and delete only on request

  Chose marking a corrected fact no longer valid and linking it to the one that replaced it, and deleting only when the visitor says "forget" or presses Forget me, over replacing facts in place because a correction ("actually I'm at Acme now") keeps the record of what the agent believed and when, while Forget me still erases the thread and everything linked to the visitor on my server.

  Cost: Old facts stay in the store until the thread has been idle for 30 days or the visitor erases it.

• VC-F

  One thread per visitor

  Chose one thread per browser visitor id, holding state, facts and an append-only journal, with every write checked against a version number, over memory per call, or one store shared by every visitor because Agents 2.0's own advice is one thread per person for one memory, and a refresh or a second tab can put two jobs on one visitor, so a write that loaded an old version re-reads and re-applies its changes instead of overwriting.

  Cost: There is no lease, so two live sessions for one visitor can still interleave turns; and the id belongs to the browser, not a login.

Outcome

This page and its /talk demo were built overnight, Oct 19 to Oct 20, while a second harness run worked on the agent to the same written contract. The agent is still being built. No latency numbers are measured yet. They go on this page once I have measured real sessions, and until then it shows none. The agent is written for livekit-agents 1.8 and only maps onto Agents 2.0: a thread per visitor, its state, an append-only journal, voice and text in one memory, a version fence where 2.0 adds a lease, and a thread meant to expire once it sits idle, as a 2.0 thread does.

Links

• ▶︎Talk to my concierge
• </>voice-concierge on GitHub (opens in new tab)