Docs / Operation reference / screen operations

screen operations#

8 operations. Generated from assets/operations.json; all areas.

screen.marks#

Mark tappable elements

Take a screenshot with numbered boxes over what can be tapped, and a list that says what each number is (label, text, description, resource id, class, bounds, centre), so a vision-capable agent can answer "tap 7" and call screen.tap. On Android the marks are the enabled clickable, long-clickable, checkable and editable elements of the UI hierarchy, in document order; content an app draws itself (a canvas, a game, much of a WebView) is not in it. On the virtual iPhone, which has no UI hierarchy, they are the lines of text Tesseract reads off the screenshot, each with its confidence (source ocr, which Android can use too): an icon without a label gets no mark, and text it misreads or misses is wrong or absent. The numbers belong to this look at the device; screen.tap checks the screen still matches before it taps. The result's path is the marked PNG, which MCP returns inline. Needs tesseract only for source ocr.

Annotations: writes, idempotent, open world.

tomaccoctl call screen.marks '{"device":"<device>","path":"<path>"}'

screen.tap#

Tap a mark

Tap the centre of a numbered mark from the last screen.marks (or screen.tree) of this device. First it checks the screen still has what the mark was made from: on Android the same element (class, resource id, text and bounds) in the UI hierarchy; for a mark read by OCR, the pixels where it was, unchanged. If not it answers mark_stale and taps nothing, so a number never lands on another element after the screen moved on. Pass duration for a long press.

Annotations: writes, not idempotent, open world.

tomaccoctl call screen.tap '{"device":"<device>","mark":1}'

screen.tree#

Read UI tree as text

Read the Android UI hierarchy as a short indented text for a model's context: only the elements that carry text or a description, can be pressed or scroll, each with its class, text, id, bounds and state (checked, disabled, focused, editable), and #N on those screen.tap can tap. The first line says the screen size and the app's package. The text stops at max_chars with a last line that says how many nodes are missing. Android only: the virtual iPhone has no UI hierarchy (ui_hierarchy_unavailable); use screen.marks with source ocr there.

Annotations: writes, idempotent, open world.

tomaccoctl call screen.tree '{"device":"<device>"}'

screen.ocr#

Read text on screen

Read the text on a device's screen or in a PNG with Tesseract, for screens that have no UI hierarchy: the virtual iPhone, games, WebViews. Answers every word and line with its box and Tesseract's confidence (0 to 100), and with text (whole words, any letter case) or text_contains (part of a word or line), the matches with their centre, ready to tap. The recognition is of pixels, not of controls: small, low-contrast or decorative text can be misread or missed, a space can be lost ("Signin"), and a picture with light and dark areas may need region. A dark theme is handled by also reading the inverted picture. Needs tesseract (capabilities.ocr says whether it is installed).

Annotations: read-only, idempotent, open world.

tomaccoctl call screen.ocr

screen.diff#

Compare two screens

Describe what changed between two screens in structured form. before and after are both UIAutomator XML files (hierarchies: elements added, removed and changed in text, state or position, or a different screen, from screen.tree or screen.marks with xml_path) or both PNG screenshots (the regions where pixels changed, each with its box, size, share of changed pixels and position, and a summary; status bar regions are marked). Leave after out and give device to compare before with the screen now. With text:true the screenshots are also read with Tesseract and the lines added and removed are listed. Screens of different sizes (a rotation) answer size_changed. before and after can also both be export bundles from screen.export (the directory or its screen.json): the answer then lists the metadata that changed (device, display, theme, locale, app), the element difference and the pixel regions.

Annotations: read-only, idempotent, open world.

tomaccoctl call screen.diff '{"before":"<before>"}'

screen.wait#

Wait for text or a steady screen

Wait until text is on the screen, is gone, or the screen has stopped changing, polling every interval_ms. text and text_gone look in the UI hierarchy's text and descriptions where there is one (Android, case-insensitive, part of a text) and otherwise read the screenshot with Tesseract; source forces one. stable takes screenshots until stable_ms pass with at most threshold of the pixels (outside ignore) changed from one to the next, and answers screen_unstable with the last changed fraction when a video, animation or spinner keeps going. A timeout answers text_timeout or screen_unstable with what was seen. The same waits are the wait_text and wait_stable steps of testing.run scenarios.

Annotations: read-only, idempotent, open world.

tomaccoctl call screen.wait '{"device":"<device>","until":"text"}'

screen.export#

Export a screen

Write one screen of a device as a bundle other tools can read: screen.png at full resolution and screen.json, a documented, versioned format (format "tomacco.screen", version 1; docs/screen-export.md and assets/schemas/screen-export-v1.schema.json). screen.json has the capture time, the device (platform, id, model, OS version, form factor), the display (pixels, density, size in dp or pt, orientation, rotation), the insets of the status bar, navigation bar and cutout where the platform reports them, the theme (dark or light), the locale, the foreground app, and every element of the screen as a flat list with ids and parent ids: role, class, text, description, resource id, bounds in pixels and in dp or pt, enabled, clickable, focusable, checked and selected, and the mark number screen.marks gives it. On Android the elements are the UI hierarchy. The virtual iPhone and USB iPhones (platform ios) have none, so there the elements are lines of text read off the screenshot with Tesseract (source ocr, state flags null) and hierarchy says unavailable. A field the platform does not report is null, never guessed. The marks are kept for screen.tap as after screen.marks. The directory must be new or empty. The result summarises the export; read screen.json for the elements.

Annotations: writes, not idempotent, open world.

tomaccoctl call screen.export '{"device":"<device>","directory":"<directory>"}'

screen.export_batch#

Export screens of several devices

Export the screen now of several devices (Android serials and virtual-iphone) in one call, each as a screen.export bundle in NN-device/ inside one directory, with index.json (format "tomacco.screen-index", version 1) listing each screen's device, model, size, orientation, theme, locale, app, element count and files. A device that cannot be read is listed with its error and the others still run; ok is false then. It takes the current screen of each device, not a matrix of languages and themes: android.store_screenshots does that for store screenshots. The directory must be new or empty.

Annotations: writes, not idempotent, open world.

tomaccoctl call screen.export_batch '{"devices":["<device>"],"directory":"<directory>"}'

View this page as Markdown on GitHub