Search what you see: Google’s AI Mode now combines Gemini and Lens for visual queries

Google is rolling out a major update to its AI Mode chatbot, introducing multimodal capabilities that allow it to process and respond to questions about images. The feature combines the company’s Gemini AI with Lens image recognition and is now available to millions more users in the US through the Labs programme.

With the new update, users can take or upload a photo and ask questions about what they see. Google says AI Mode will return a "rich, comprehensive response with links" to explore further, all powered by a custom version of Gemini built specifically for search.

"AI Mode builds on our years of work on visual search and takes it a step further," said Robby Stein, VP of product for Google Search. "With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colours, shapes, and arrangements."

The feature uses what Google calls a "fan-out technique", issuing multiple simultaneous queries about both the image as a whole and the individual objects within it. The company says this approach helps deliver results that are more nuanced and contextually relevant than traditional search, such as identifying books in a photo and offering suggestions for similar titles with positive reviews.

Google says it has seen strong engagement from early AI Mode users, who are asking longer, more complex queries than those typically seen in Search. People are reportedly using the feature for tasks like comparing products, planning trips and exploring how-tos.

AI Mode was initially available only to Google One AI Premium subscribers through Labs, but with this update, access is expanding to a wider user base in the US. The company has not yet confirmed when the feature will be available in other regions.

The multimodal update is live now and can be accessed through the Google app on both Android and iOS.