Microsoft just taught AI to click, type, and navigate like a human being

Microsoft has introduced a powerful new feature in Copilot Studio, called computer use, allowing AI agents to autonomously interact with websites and desktop applications, mimicking the way a human would click buttons, type in fields, and navigate menus. The capability, now available in research preview for select users, empowers organisations to build intelligent agents that can perform complex tasks across both browser and desktop environments, even when no API is available.

With computer use, users can simply describe the task they want the agent to perform using natural language. The agent then simulates the action, allowing for testing and refinement before deployment. Once configured, these AI agents can automate workflows in web browsers such as Microsoft Edge, Google Chrome, and Mozilla Firefox, as well as native desktop applications.

“If a person can use the app, the agent can too,” said Charles Lamanna, Corporate Vice President of Microsoft’s Business and Industry Copilot. “Computer use enables agents to interact with websites and desktop apps by clicking buttons, selecting menus, and typing into fields on the screen.”

Designed to handle real-world business scenarios, the new feature allows for task automation such as large-scale data entry, market research, and invoice processing. Microsoft highlighted how enterprises could leverage this tool to input data from various sources into centralised systems, streamlining operations and minimising errors.

Unlike some existing agent tools that require human intervention when facing interface changes or CAPTCHA prompts, Microsoft’s computer use boasts built-in reasoning capabilities. These enable AI agents to autonomously adapt when screen elements change, ensuring tasks continue without disruption. Users also have access to a detailed activity history, including screenshots and reasoning logs for transparency and oversight.

Security and privacy remain key considerations. Microsoft has confirmed that enterprise data remains within Microsoft Cloud boundaries and will not be used to train its Frontier models. Additionally, since the feature runs entirely on Microsoft-hosted infrastructure, organisations can benefit without the burden of managing their own servers, helping to accelerate deployment while reducing maintenance and infrastructure costs.

Early access to the computer use feature is now available for Copilot Studio users, with broader rollout expected in the near future.