How an Agentic Routines Replay Checks the Page Before Every Step

A replay with no model has one job that a model usually does without being asked: noticing that something is off. If a recorded click lands on a page that has moved on, a script will click anyway. This post is about the checks Agentic Routines runs before every step so that it doesn’t, and what happens when one fails. The first post is the overview.

Three checks, cheapest first

Before each step the replayer checks the page in a fixed order and stops at the first check that settles the question.

1. Page structure. The address pattern, plus the page’s headings, form labels and main regions. It costs almost nothing and catches the common failure early: the routine is on the wrong page.
2. The element. Its role and name, its visible text, stable attributes and its place on the page. It must match exactly one element. Zero or two are both a failure.
3. A screenshot crop. The area around the element is compared to the one taken during recording. Password fields are always masked in that crop.

step 4 of 6: click "Next"
  page      ok    /apps/*/releases/new, heading "Create internal testing release"
  element   ok    button "Next", one match
  picture   ok    crop matches the recording

The order goes from cheap and broad to specific. A renamed button or a different page shows up in the first two checks without a picture. The crop is there for what those two can miss: the right element in a place that has changed meaning, such as the same “Save” button inside a different dialog.

When a step has changed

If the element is missing or matches more than one, the replay does not guess. It pauses and hands the page to the agent that is already running, with the candidates listed. The agent picks the right one, and that choice is saved for next time. If the agent cannot pick, it finishes the task by hand and that run is saved as the next version.

step 4 of 6: click "Next"
  page      ok
  element   no match for button "Next"
  -> paused, handed to the agent with the candidates on the page
     ref_41 button "Continue"
     ref_44 button "Back"

That is where the numbers in the overview come from. When I renamed one button after recording, the long release task still finished at $0.08 instead of $0.06, because the model was involved for one step and not for six.

There is one rule I was strict about. A replay never follows text from the page. It runs the recorded steps, and the only thing that can choose a different element is the agent’s or your answer, by reference. A page cannot argue its way into a click. The next post is about that.

Making the recording shorter

The checks matter more when a routine is short, because each recorded step is another place to fail. After a task the agent turns the run into a draft where every step is numbered, and the draft ends with hints. A hint is a suggested edit, never applied by itself. The one I use most is navigation only: a run of clicks that only got from one page to another, followed by a step at a known address.

before   click Apps -> click "Acme Console" -> click Releases -> click Create release
after    navigate https://console.example.com/apps/{{app}}/releases/new

A real case is releasing a build to a testing track in a store console. Publishing shows a dialog that offers to go back to the overview. The recorded run clicked through that dialog and then clicked its way to the page the next step needed. Replacing it with one address makes the routine shorter and takes the dialog out of the picture. The other hints cover dialogs that were only answered, repeated clicks, steps that submit and steps on another site.

The agent applies edits with routine_edit, replays the draft once with routine_test, and only then saves. The test never performs a step that pauses, so nothing irreversible runs while it is checking.

What an edit cannot do

I did not want the safety of a routine to depend on the agent editing it well, so the extension refuses some edits:

• A step that submits, publishes, pays or deletes cannot be removed or replaced, and its pause cannot be switched off. The agent can edit the steps around it.
• An address has to stay on the routine’s own site, over https unless the recorded page was http.
• An address cannot carry a per-session value. Long random strings, UUIDs and parameters named like sid, token or state are refused. The part that changes becomes a parameter the routine asks for.
• The result has to pass the rules the directory applies when a routine is published, so a routine that would be turned away is caught on the way in.

Window size

A routine records the size of the page it was recorded on. A replay compares it with the page it finds. In the same responsive band, nothing changes. One band over and close in width, the replay goes on as usual. Very far apart, say a phone-sized window for a routine recorded on a desktop, the layout may differ, so the result says so and the picture check is switched off for that run. The page structure and the element decide, and the stored picture is left as it was. match_viewport can resize the window to the recorded size first.

Where I expect it to break

• A site that changes its structure and its labels at once will fail all three checks, and the agent will do that step by hand. That is the intended behaviour, but it means a very volatile site gains little.
• Pages that build their content late are a risk I have not measured. The replay can look at a page that is not finished yet.
• The benchmark runs were on one console replica and one real site. I have not measured how often real sites change under a routine.

The docs are at agenticroutines.arunprakashg.com/docs. The repository is private for now and opens soon.