Docs / Operation reference / testing operations

testing operations#

16 operations. Generated from assets/operations.json; all areas.

testing.find#

Find Android element

Wait for native UI elements using semantic properties; return bounds and attributes. The virtual iPhone (device virtual-iphone) reports no UI hierarchy, so a selector with only text or text_contains (and index) there is read off the screenshot with Tesseract instead: the matches carry source ocr, pixel_based true and a confidence, and the text can be misread or missed. Any other selector property answers ui_hierarchy_unavailable, whose details.testing lists what works there.

Annotations: read-only, idempotent, open world.

tomaccoctl call testing.find '{"device":"<device>","selector":{"text":"<text>"}}'

testing.action#

Operate Android element

Click, long-click or fill a uniquely matched native Android element. The virtual iPhone (device virtual-iphone) reports no UI hierarchy, so there a selector with only text or text_contains (and index) clicks or long-clicks the middle of the text read off the screenshot with Tesseract (source ocr, pixel_based true); fill and any other selector property answer an error, and ui_hierarchy_unavailable lists in details.testing what works.

Annotations: writes, not idempotent, open world.

tomaccoctl call testing.action '{"device":"<device>","selector":{"text":"<text>"},"action":"click"}'

testing.run#

Run native tests

Run JSON native UI scenarios (one file, or a directory of them) across devices with device locks, JUnit, JSON reports, failure screenshots and a self-contained report.html; each run's outcomes are added to the test history (testing.history). A device is an Android serial or virtual-iphone, the running virtual iPhone. By default every device runs every scenario; shard splits the scenarios across the devices and merges their results into one report. With retries, a failed scenario runs again; each result lists its attempts and flaky marks a pass after a failure. Besides selector steps, a scenario can tap and swipe at pixel coordinates, type text, send a key, pause, take a screenshot, assert_screen (compare the screen with a baseline PNG), wait_screen (until it matches), wait_text (until text is on the screen, or with state absent gone; from the UI hierarchy on Android, read with Tesseract where there is none or source is ocr) and wait_stable (until the screen has not changed for duration milliseconds). On Android, a measure step records startup, frames, memory or usage metrics of an app, and a budget fails the scenario as budget_exceeded when a metric is outside it. On the virtual iPhone, steps go through ios.emulator.control at pixels of ios.virtual.screenshot, launch opens an app through the guest, and stop, clear and measure fail, and so do selector steps other than a text or text_contains selector, which is read off the screenshot with Tesseract (pixel-based; wait and assert, click and long_click, not fill): the guest reports no UI hierarchy. Its crash reports are not read after a scenario; use testing.crashes. With platform ios, it runs the same scenarios on USB iPhones through the WebDriverAgent session runtime.start keeps for each device (start it first): selectors are read against WDA's XML source, visible elements only, with text as the label (or the value of an element without one), resource_id as the accessibility identifier, description as the label and class as the element type; tap and swipe coordinates are in points; keys are home, enter, backspace, delete or tab. clickable, checked and package selectors and the clear step are refused there, and crashes are not read.

Annotations: writes, not idempotent, open world.

tomaccoctl call testing.run '{"path":"<path>","directory":"<directory>","devices":["<device>"]}'

testing.record#

Record UI test

Record a UI test on an Android device or the virtual iPhone. Between start and stop, the taps, swipes, text and keys sent to the device through android.input, runtime.input (the desktop's live display included) and testing.action become testing.run scenario steps. A tap becomes a click on the element under it, named by resource ID, description or text (with index when that is shared), when the UI hierarchy read just before the tap names one; otherwise a tap at pixel coordinates. Reading the hierarchy makes each tap slower while recording. stop writes the scenario to path; status counts the steps; discard throws the recording away. Input a scenario cannot express, such as a rotation, is listed under skipped. With device virtual-iphone (while it runs), start notes the screen's size and the ios.emulator.control input sent to its guest job becomes steps at pixel coordinates: click a tap, drag a swipe, a streamed touch_down to touch_up a tap, long press or swipe, and text, key and home their steps; a key combination and power_button are skipped.

Annotations: writes, not idempotent, closed world.

tomaccoctl call testing.record '{"device":"<device>","action":"start"}'

testing.lint#

Check UI test scenario

Check a testing.run scenario without acting on the device: its format, and each step against the current Android screen (or a saved android.hierarchy XML). findings name the step and a code: not_found (no element here matches; a warning, since an earlier step may open the screen it is on), ambiguous (several match: an error for click, long_click and fill, which need one), disabled, present (an element expected absent is there), off_screen (a tap or swipe point outside the screen) and invalid_key. steps lists each step's match count. A scenario that does not parse answers valid false with the error. passed is false when any finding is an error. The virtual iPhone (device virtual-iphone) reports no UI hierarchy: it answers ui_hierarchy_unavailable, whose details.testing lists what works there.

Annotations: read-only, idempotent, open world.

tomaccoctl call testing.lint '{"path":"<path>"}'

testing.visual_diff#

Compare with baseline image

Compare a PNG screenshot with a baseline PNG pixel by pixel. Reports changed_fraction, changed_bounds and passed (changed_fraction <= threshold); writes a diff image with changes in red when diff is given. update_baseline keeps the actual image as the new baseline instead of comparing.

Annotations: writes, idempotent, closed world.

tomaccoctl call testing.visual_diff '{"actual":"<actual>","baseline":"<baseline>"}'

testing.accessibility#

Check Android accessibility

Audit an Android screen's UI hierarchy: clickable elements without a label (missing_label), touch targets under min_target_dp (small_target) and text shorter than min_text_height_dp (small_text). Returns structured findings with bounds in pixels and sizes in dp. Heuristics, not a certificate: no contrast, font size or focus order. The virtual iPhone (device virtual-iphone) reports no UI hierarchy: it answers ui_hierarchy_unavailable, whose details.testing lists what works there.

Annotations: read-only, idempotent, open world.

tomaccoctl call testing.accessibility

testing.crashes#

Find app crashes

Read logcat's crash, system and main buffers and return Java crashes (FATAL EXCEPTION), ANRs and native crashes (Fatal signal) as structured events with package, pid, exception, message, stack and reason. Window: since (epoch seconds on the device clock), the life of job_id, or the last lines lines. With device virtual-iphone, reads the running virtual iPhone's crash reports whose names start with process through the guest service, as ios.virtual.crash_reports does, and returns each one with an exception as a crash event: package (bundle identifier), process, pid, thread, exception, signal, message, termination and the crashed thread's frames as stack, timed by the report's capture time; unparsed lists the rest with the reason. The guest returns at most three reports per directory, so collection_complete false means some were left out, and an empty list never proves that nothing crashed.

Annotations: read-only, idempotent, open world.

tomaccoctl call testing.crashes '{"device":"<device>"}'

testing.results#

Read test results

Read the JUnit XML results of a project's tests, or the JSON reporter output of Dart and Flutter tests, into one summary: totals, suites and each failure with its message and stack. Finds Gradle unit (build/test-results) and instrumented (build/outputs/androidTest-results) results and the Flutter reports project.execute writes to build/tomacco, or reads a given directory or file such as testing.run's junit.xml or a flutter test --machine output. golden_failures lists each failed Flutter golden-file comparison with its message and the masterImage, testImage, isolatedDiff and maskedDiff images flutter_test wrote. since or job_id leaves out results older than the run.

Annotations: read-only, idempotent, closed world.

tomaccoctl call testing.results '{"path":"<path>"}'

testing.coverage#

Read code coverage

Read the JaCoCo XML coverage reports of a project's tests into totals (instruction, branch, line, complexity, method, class: covered, missed, percent), each source file with its missed lines, and each class, least covered first. Finds the reports AGP writes for unit tests (enableUnitTestCoverage) and instrumented tests (enableAndroidTestCoverage) under build/reports/coverage, the jacoco plugin's under build/reports/jacoco and Kover's XML under build/reports/kover; project.execute with coverage:true runs the tasks. Several reports are merged line by line: a line is covered when any report covers it, and each class comes from the report that covers it most. baseline, a JSON report an earlier call wrote or JaCoCo XML, adds delta: the change of each total and each file whose line coverage moved, appeared or went away. minimum and max_drop are checks on line coverage that set passed.

Annotations: read-only, idempotent, closed world.

tomaccoctl call testing.coverage '{"path":"<path>"}'

testing.report#

Write HTML test report

Write one self-contained HTML file for a test run: for a testing.run report directory, each scenario and device with its outcome, attempts, steps and their timings, screenshots, baselines and difference images, crashes and metrics; for a project, a results directory or file, the JUnit and Dart JSON results with totals, suites, every test case and each failure with its stack and Flutter golden images. Styles are inline, images embedded as data URIs (up to 48 MiB in all, beyond that linked by path) and there are no scripts. testing.run and project.execute test jobs write theirs when they end; call this for another place or an earlier run. Returns the file as html with what it holds.

Annotations: writes, idempotent, closed world.

tomaccoctl call testing.report '{"path":"<path>"}'

testing.history#

Test pass-rate history

Pass rates of tests and UI scenarios across runs, to find flaky ones. testing.run adds each scenario's outcome per device when it ends and a project.execute test or instrumented-test job each test case, to a store in Tomacco's data directory; record adds a testing.run directory, a project's or directory's current JUnit or Dart JSON results, or a file, once per file version. summary lists each test with runs, passed, failed, skipped, passed_after_retry, pass_rate (percent of runs that passed, retried passes included), flips (changes between pass and fail), recent (P passed, R passed after a retry, F failed, S skipped; newest last), last_failure and flaky (both outcomes in the window, or a retried pass), flaky tests first, the least reliable first. last limits each test to its newest runs.

Annotations: writes, idempotent, closed world.

tomaccoctl call testing.history

testing.impact#

Select affected tests

Pick the tests a change likely affects in a Gradle build, from the changed files: git diff against base (HEAD by default, so uncommitted and untracked changes) or files. A changed test class is selected itself; a changed Kotlin or Java source selects the unit and instrumented test classes of its module and of the modules that depend on it (project(...) and projects.x in their build files) that are named after it (NameTest, NameTests, NameIT, NameSpec, TestName) or use its name in the same package or with an import, and otherwise the tests of its package in its module. Resources or the manifest select the module's instrumented tests, a module's build file all its tests and its dependents', and the settings, versions catalog, gradle.properties, gradle/, buildSrc or build-logic everything (run_all). scenarios, a testing.run scenario file or directory, adds the scenarios that launch an app whose module a change touches. commands are the Gradle calls with --tests and the instrumentation runner's class argument; run unit or instrumented starts them as a job. The answer lists unmapped files. Heuristics over the files: Gradle is not run, and a test reached through reflection, dependency injection or generated code can be missed.

Annotations: writes, not idempotent, open world.

tomaccoctl call testing.impact '{"path":"<path>"}'

testing.baselines#

Review screenshot baselines

Manage the baselines of assert_screen and wait_screen steps. review lists each screen step of a testing.run report directory per device as new (no baseline yet), changed, size_changed, passed, failed or not_reached, with its id, baseline, screenshot (actual), difference image and changed_fraction; by default only the screens to review (new, changed, size_changed), all_screens lists every one. approve copies the screenshots of the named screens, or with all of every new and changed screen, over their baselines, and refuses before writing when two devices would write different images to one baseline. list shows the PNG baselines in a directory with their sizes. Pair it with testing.run's baselines directory and record_baselines.

Annotations: destructive, idempotent, closed world.

tomaccoctl call testing.baselines

testing.setup#

Set up Android text input

Build and install the native Android instrumentation for Unicode text replacement. Returns a durable job.

Annotations: writes, idempotent, open world.

tomaccoctl call testing.setup '{"device":"<device>"}'

testing.instrumentation#

Check Android text input

Check the installed native Android instrumentation protocol.

Annotations: read-only, idempotent, open world.

tomaccoctl call testing.instrumentation '{"device":"<device>"}'

View this page as Markdown on GitHub