Steps and replay¶

Agents that browse, run commands or edit files are easier to trust when the user can watch them work, as in Manus's "computer" view. AI Elements shows each step for what it is and keeps a live preview while the agent runs. The whole view expands into a panel with a timeline. Any run can be stepped through afterwards, and runs with a recorded event log can be replayed event by event.

What a step does: tool categories¶

A tool call carries two model fields, ToolPart.category and ToolPart.location. The category uses the Agent Client Protocol's ToolKind vocabulary: read, edit, delete, move, search, execute, think, fetch, switch_mode and other. The location is the path or URL the call acts on. Like every model field, both are set upstream, never guessed by the UI:

val listFiles = object : AgentTool {
    override val name = "list_files"
    override val description = "List the files in a folder."
    override val parameters = buildJsonObject { put("type", "object") }
    // What the call does (ACP ToolKind) and where: the computer shows it as a search of that folder.
    override fun categoryFor(arguments: JsonObject) = ToolCategory.SEARCH
    override fun locationFor(arguments: JsonObject) = arguments["path"]?.jsonPrimitive?.contentOrNull
    override suspend fun execute(arguments: JsonObject) = "README.md\nsrc/"
}

Screenshots of each step¶

A tool call's screenshots are the image files that follow it in the reply. Each protocol has a standard way to carry them, and no extension is needed:

In the conversation, a step's screenshots show as a small strip under its tool call. Tapping one opens the agent's computer at that step. If you build your own view, agentSteps(message) returns the steps of a reply: each tool call with its image files.

The agent's computer¶

A reply whose steps have a category or a screenshot gets a live preview card (AgentComputerCard). The card shows:

• the latest screenshot, or an icon for what the current step does;
• what the agent is doing, such as "Running command · ls -la" or "Reading · src/App.kt";
• the number of steps;
• a live marker while the reply streams.

Tapping the card opens AgentComputerPanel. It shows the selected step by its category, reusing the existing elements:

Below the view the panel has:

• the step's status;
• a timeline slider;
• previous, play and next buttons;
• a chip for every step.

While the reply streams, the panel follows the newest step. Picking an earlier step stops following, and Back to live resumes it.

The layout adapts to the space the chat actually has (AgentComputerScaffold). At 720 dp or wider, the panel is a side pane next to the conversation, as in Material 3's supporting-pane layout. This also works inside a list–detail layout, because the width checked is the chat's own, not the window's. When narrower, the panel is a ModalBottomSheet. Chat and Conversation set this up. To share one panel across your own layout, create a state with rememberAgentComputerState and wrap your layout in AgentComputerScaffold (layout: Auto, SidePane, BottomSheet).

An app that already uses Material 3's adaptive ListDetailPaneScaffold for its conversation list should host the panel in the scaffold's extra pane (layout = AgentComputerLayout.Hosted, and AgentComputerPanel in extraPane). This is the canonical list–detail–extra layout, and the scaffold then decides:

• On expanded windows (tablets, unfolded foldables), the computer opens on the right, beside the chat, and the history list steps aside.
• In a single pane, the computer opens over the chat.
• Back returns to the list and the chat.

The demo app does this: open the computer on navigation to the extra pane, and close it when the user navigates away.

Stepping through works on any saved conversation, because it only reads the stored message.

Replay¶

Replay has two levels.

• Steps, from the messages. This is how the AI SDK persists chats: UIMessage[], as useChat stores them in onFinish. It is also how AG-UI's MESSAGES_SNAPSHOT carries them. The panel above needs nothing more, and it works for every backend.
• Events, from AG-UI's event log. This is the standard for event-level fidelity: the timing, every delta and every state change.

AG-UI event logs¶

AG-UI serialization defines the log. Give AgUiBackend an eventLog and it keeps one:

• Events as streamed. One append-only log per threadId, stored as a JSON array of events. Events without a timestamp get the time they arrived; BaseEvent.timestamp is the spec's own field.
• Runs linked by parentRunId. Each run names the one before it, in both RunAgentInput and the logged RUN_STARTED. This is the spec's branching lineage.
• What the client added, in RUN_STARTED.input. When the server did not echo the input, the log records it there, keeping only messages not already in the log, as the spec's compaction rule says. This matters for frontend tools: their results exist only on the client.
• The runs that produced a reply, in its metadata. They are stored under the agui key of Message.metadata, so a stored conversation still knows which runs to replay.

AgUiEventLog.replayOf(message) plays a reply's runs back through the same AG-UI event parser the live stream used, at the recorded pace, and rebuilds the reply as it happened. Pass it to Chat, and the panel offers Replay run:

// AG-UI serialization: every run's events, one JSON log per thread.
val log = remember { AgUiEventLog.Files(File(context.filesDir, "agui-events")) }
val controller = rememberChat { approver ->
    AgUiBackend("https://agents.example.com/api/agui", approver = approver, eventLog = log)
}
// The agent's computer offers "Replay run" for replies with recorded events.
Chat(controller, replay = { message -> log.replayOf(message) })

AgUiEventLog.Files keeps one file per thread; AgUiEventLog.InMemory keeps the log for the life of the process. Implement the interface to store logs elsewhere, for example in your own database or on your server. Screenshots make logs large, so prefer url sources to inline data for runs you keep.

ACP: the agent's own record¶

An ACP agent that can load sessions (loadSession) replays a session on session/load: it streams the whole conversation back as session/updates. Pydantic AI Harness does this when the adapter has a SessionStore. AcpBackend.replayOf(message) does the following:

1. It opens a connection of its own, so the session you are chatting in is left alone.
2. It loads the session.
3. It takes the reply's turn (its metadata records which turn it was).
4. It plays that turn through the same mapping as a live turn.

ACP updates carry no timestamps, so a replay plays at a steady pace. An agent that runs as a local process only has sessions its store keeps across processes.

Compacting logs¶

AgUiEventLog.compact implements AG-UI's compaction, ported from the reference compactEvents:

• a message's or tool call's deltas become one event;
• each run's state becomes one STATE_SNAPSHOT;
• events that arrived mid-stream move after that stream.

Compaction reorders events, so compact a thread when you archive it. Keep the recorded log for as long as you want event-level replay.

Other protocols¶

• AI SDK. The AI SDK has no client-side event log. resumeStream continues an in-flight response from the server's stream store; it does not replay one. Use message-level replay.
• On-device agents. Their replies are stored as messages; use message-level replay.
• Server side. Pydantic AI Harness StepPersistence keeps an agent's append-only step log for recovering, continuing and forking runs.
• Traces. For debugging and evaluation, use OpenTelemetry's GenAI semantic conventions (Pydantic Logfire, Langfuse and similar).

See Choose your setup for which of these the library provides and which your app decides.