Google has launched Gemini 3.7 Flash, a hybrid reasoning model designed for coding and agentic workflows, demonstrating its capability to control Android devices through visual input. The article details how the model inspects raw screenshots to plan actions and emit precise coordinates, enabling interactive device control loops without relying on accessibility trees.
- The agent loop captures screenshots via ADB, evaluates them with Gemini 3.7 Flash, and emits tool calls with normalized 0–999 coordinates.
- A Python script maps these coordinates to physical pixels and executes actions like taps and text input.
- In a test case, the model successfully navigated Chrome, cleared popups, and solved Wordle in two guesses by reading color-coded feedback.
- This approach works across native apps, webviews, and dynamic canvas interfaces where traditional automation tools fail due to missing accessibility nodes.
The open-source quickstart repository allows users to build automated UI testing, user flow verification, and exploratory bug reproduction tools that function regardless of underlying DOM structures.