Turn any LLM into a Computer Use Agent
OmniParser ‘tokenizes’ UI screenshots from pixel spaces into structured elements in the screenshot that are interpretable by LLMs. This enables the LLMs to do retrieval based next action prediction given a set of parsed interactable elements.
OmniParser V2's icon caption model is trained with Responsible AI data to avoid inferring sensitive attributes (race, religion, etc.) from images. The system includes threat modeling, sandbox containers, and safety guidance, enabling organizations to deploy GUI agents aligned with Microsoft AI principles and ethical AI practices.
OmniTool, the dockerized Windows system bundled with OmniParser V2, provides out-of-the-box integration with multiple state-of-the-art LLMs and includes essential tools for screen understanding, grounding, action planning, and execution—allowing developers to experiment with different agent settings without building infrastructure from scratch.
OmniParser V2 achieves 39.6% accuracy on ScreenSpot Pro (vs. GPT-4o's 0.8% baseline) through larger training datasets for interactive element detection and icon captioning. It reduces latency by 60% versus V1, enabling faster, more reliable detection of tiny interactable elements that general LLMs struggle to locate.
OmniParser V2 converts UI screenshots into structured, LLM-interpretable elements by tokenizing pixels into interactable components. This allows any LLM (GPT-4o, DeepSeek, Qwen, Anthropic) to reliably identify buttons, icons, and UI regions, then predict and execute the next action—solving the core challenge of GUI automation without requiring models to process raw pixel data.