Google’s next AI technology could control computers

Google might be working on artificial intelligence technology that could eventually control computers via a web browser interface capable of performing tasks such as research, shopping, or flight bookings. According to a recent report, this technology, which is internally referred to as 'Jarvis,' will be built on an upcoming version of Google’s Gemini large language model and could be released as early as December.

According to a report by The Information, citing people with direct knowledge of the product, Jarvis will likely be available in the Google Chrome browser and help users “automate everyday, web-based tasks.” The technology could take screenshots of the current web browser activity, understand the content and context, and then perform the necessary actions by either clicking buttons or entering text automatically. The current prototype, according to the report, takes a few seconds to perform the sequence of actions.

Jarvis will be Google’s answer to Rabbit’s large action model, but it could also be part of the company’s strategy to stay ahead of the competition. For instance, Microsoft’s Copilot Vision lets Windows users talk to the AI chatbot about the web pages they are viewing. They could interact with the bot about a topic, but the company has not said whether the Copilot Vision will be able to automate tasks like Jarvis. Apple , which comes across as a late entrant to the AI space, has also announced that the upcoming Apple Intelligence will be able to read screens and offer actions based on the content. Anthropic recently announced a new beta update to Claude, which it said can use a computer on behalf of users. OpenAI is also coming up with a similar version, which could be integrated into ChatGPT for automating complex tasks “by effectively taking over a customer’s device.”

While the report has anticipated that Jarvis could be announced as early as December, Google may change its plan based on how far the progress on the technology would have reached. Google is also reportedly planning to release the technology to a few testers who could help it discover and fix bugs in Jarvis.