Two clocks: running slow reasoning inside a fast conversation

You put an LLM into something live: a voice agent, a copilot, a tutor, a realtime chat. It works in the demo. Then you add the thing that makes it actually good, real per-turn reasoning, scoring the answer, deciding what matters next, pulling context, and suddenly every reply lands a beat too late. The pause is small. It is also fatal, because a live conversation lives or dies on rhythm, and a thinking pause reads as a broken bot, not a careful one.

I hit this building a realtime voice AI that interviews people and has to reason hard about every answer while still feeling like a calm human on the line. The instinct is to make one model do both jobs: talk naturally and think rigorously, in one pass. That is the trap. The way out is to stop asking one clock to do two speeds.

The split: a fast clock and a slow clock

The move is to run two clocks at once and let neither block the other.

The fast clock gates the reply. Before the agent speaks, the only thing it waits on is a single cheap classifier doing the bare minimum needed to not be rude: did the person actually finish, are they still talking, are they fixing what they just said, are they asking me something back. That is enough to take a turn without interrupting or double-asking. Nothing expensive runs here.

The slow clock runs everything else, off the reply path. All the real reasoning, the heavy scoring, the analysis, finding the highest-value gap, the retrieval, is fired off concurrently and is not allowed to hold up the spoken reply. Crucially, its result is not consumed by this turn. It is consumed as context for the next turn.

That last sentence is the whole pattern. You let the agent reason against a picture that is one turn stale, and you accept that on purpose.

Why one turn stale is fine

It feels wrong to ship an agent that is always reasoning against slightly old information. But walk through what the slow result is actually for. It is not grading the answer you just heard for its own sake. Its job is to shape the next question. And the next question does not exist yet. So there is a free window, the time the person spends talking, in which the slow clock can finish and have its answer pinned and ready before it is ever needed. Being a turn behind costs you nothing if the thing you compute is consumed a turn later anyway.

The reframe that matters: this looks like a performance problem ("reasoning is too slow for realtime") and it is actually a design decision ("reasoning belongs on a different clock than speech"). Once you see it that way, you stop trying to make the model think faster and start arranging when each kind of thinking is allowed to happen.

Take turn-taking away from the model

There is a corollary that surprised me, and it is non-negotiable once you commit to two clocks. The talking model cannot be allowed to decide when it speaks. In my build the voice model is muted by default: it is structurally forbidden from starting a turn on its own. A separate, deterministic controller is the sole turn-taker, and it only lets the model speak once the slow clock's next question is ready.

This is plain server logic, not a clever model deciding what to do. If you let the generative model run the conversation, the two clocks immediately fight: the model fires a reply on its own timing, the slow clock's result arrives a moment later with nowhere to go, and your careful pipeline becomes decorative. Whoever owns turn-taking owns the rhythm, and it has to be your code.

Most of the hard work is refereeing collisions

Here is what nobody warns you about. Once two clocks share a live human, the bulk of your difficult code is not the reasoning or the speaking. It is handling the races between them, and none of it shows up on a good day:

• The go-around. The person keeps talking while a reasoning pass is mid-flight. If you are not careful, the agent advances to the next topic based on a question it never actually asked. The fix is to quietly discard the pending move and stay on what the person is still saying.
• Coalescing fragmented input. Real speech is stop-start. You need a short quiet-timer that gathers fragments into one real turn before anything reacts, or the agent answers half a sentence.
• Suppressing unsolicited output. A generative model will sometimes try to emit a turn you did not ask for. You need to recognize and drop those by id, or the muting you carefully designed leaks.

If you sketch this on a whiteboard it looks trivial. In production it is most of the build. Budget for it.

When NOT to split into two clocks

This is machinery, and machinery has a cost. Do not reach for it when:

• The interaction is turn-based and patient. A chat where a two-second pause is normal does not need this. Just reason inline and show a thinking state. Two clocks earn their keep only when the medium punishes pauses: voice, live tutoring, anything that feels broken when it hesitates.
• Your per-turn reasoning is already fast enough. If the heavy step fits inside your latency budget, you do not have a problem to solve. Adding a second clock to a system that does not need one is the same over-engineering trap as a five-agent orchestrator for a one-shot task.
• There is no useful 'next turn' to pre-compute. The one-turn-stale trick only works because the slow result feeds the next interaction. In a one-shot request with no follow-up, there is nowhere to hide the latency, so hiding it is not an option.

Adopting it: a short checklist

1. Find the cheapest possible classifier that decides whether to take a turn at all. That, and only that, gates the reply.
2. Move every expensive step off the reply path and run it concurrently.
3. Wire the slow result into the next turn's context, not this one. Accept the one-turn lag explicitly.
4. Make turn-taking deterministic server logic. Mute the generative model by default; let your code decide when it speaks.
5. Budget real time for the collision cases: the go-around, fragmented speech, unsolicited output. That is where the work actually is.

This is the timing-and-concurrency cousin of a point I have made before: you usually don't need a multi-agent orchestrator. Same spirit, different axis. There it was about not adding agents; here the turn-taker is deliberately deterministic server logic, not an LLM deciding what to do next. One capable pipeline, two clocks, beats a committee of models every time.

The one-line version

When an agent must feel fast and think hard, don't make one model do both. Gate the reply on a cheap classifier, run the real reasoning on a second clock, and feed its result to the next turn, not this one. One turn stale is the price of feeling human, and it is almost always worth paying.