Google has begun rolling out new artificial intelligence features to its Gemini Live platform, allowing it to visually interpret smartphone screens and camera feeds in real-time. The features, officially confirmed by Google spokesperson Alex Joseph, are part of the company’s broader AI initiative known as “Project Astra.”
The newly introduced capabilities include screen-reading and live video interpretation, allowing Gemini Live to answer user queries about what’s displayed on their phone screen or through their camera lens. The rollout is exclusive to Gemini Advanced Subscribers under the Google One AI Premium plan, with availability gradually expanding throughout the month.
According to Joseph, the screen-reading feature enables users to ask Gemini questions about any content visible on their smartphone screen, offering contextual responses. Meanwhile, the live video feature leverages a smartphone’s camera to provide real-time analysis of whatever is being viewed. For instance, users can ask Gemini to identify objects, suggest aesthetic decisions, or even guide them through tasks like choosing a paint colour for freshly-glazed pottery.
The first public demonstration of the screen-reading capability was reported by a Reddit user with a Xiaomi phone, later confirmed by 9to5Google. In a video shared by the user, Gemini successfully read the phone’s screen content and responded accurately to questions about it.
Google’s rollout comes as its competitors scramble to catch up. Amazon is preparing to launch its Alexa Plus upgrade with similar capabilities but remains in early access. Apple, meanwhile, has delayed the release of its revamped Siri, which is also expected to offer enhanced AI functionalities.
Samsung continues to rely on its Bixby assistant, but Gemini’s seamless integration into its phones gives Google a distinct advantage.
First teased nearly a year ago, Project Astra represents Google’s effort to redefine the boundaries of what digital assistants can accomplish. By combining visual analysis with natural language processing, Google aims to create a more interactive and intuitive AI experience.