DeepMind unveils magic pointer demo for Googlebook, coming to Gemini in Chrome

At a glance:

• AI‑enabled pointer demo available in Google AI Studio, letting users interact with images and maps by simply pointing
• Core use cases demonstrated: PDF summarisation, converting tables to charts, scaling recipe ingredients, and turning video frames into booking links
• Gemini integration in Chrome is rolling out, so the pointer can be used to query web content directly

What the magic pointer does

DeepMind’s research team built a new interaction layer they call the Magic Pointer on the Googlebook platform. The core idea is to let an AI understand not just the visual object under the cursor but also why that object matters to the user. By capturing visual, semantic and spoken context, the pointer can translate a simple gesture into a complex request without the user having to type a long prompt. The team explains that typical AI tools sit in their own window, forcing users to drag information into them. The Magic Pointer flips that model: the AI meets the user wherever they are working—inside PDFs, spreadsheets, maps or video frames—so the workflow stays uninterrupted.

Demo capabilities in Google AI Studio

Google AI Studio currently hosts two live demos of the Magic Pointer:

• Edit an image – point at any part of a picture and ask the model to modify it (e.g., change colors, remove objects).
• Find places on the map – hover over a location and request directions, nearby attractions, or travel times. These demos showcase how the pointer captures both the visual element and the user’s intent, turning a click‑and‑ask interaction into an immediate, context‑aware response.

Real‑world use‑case examples

The research paper lists several concrete scenarios that illustrate the pointer’s versatility:

1. Point at a PDF paragraph and ask for a bullet‑point summary that can be pasted directly into an email.
2. Hover over a table of statistics and request a pie‑chart version of the data.
3. Highlight a recipe and ask the system to double all the ingredient quantities.
4. Pause a travel‑video frame and have the pointer turn it into a booking link for the featured restaurant. These examples underline the shift from “text‑heavy prompts” to natural, shorthand interactions that feel like a conversation with the screen.

Integration with Gemini in Chrome

DeepMind also announced that the Magic Pointer will soon be usable with Gemini inside the Chrome browser. During the rollout, users will be able to point at any part of a webpage and ask Gemini a follow‑up question about that specific element. For instance, selecting a handful of products on an e‑commerce page could trigger a side‑by‑side comparison, or pointing at a living‑room scene could ask the model to visualise a new couch in that space. The feature is currently in a phased roll‑out, meaning early adopters will see the pointer icon appear next to the cursor when Gemini is active. As the capability expands, it promises to blur the line between browsing and interactive AI assistance, making web research feel more like a dialogue than a series of clicks.

Why it matters for productivity

If the Magic Pointer lives up to its promise, it could reshape how professionals interact with digital content. Designers could edit assets on‑the‑fly, analysts could generate visualisations without leaving spreadsheets, and everyday users could get instant answers from web pages without copy‑pasting text. The technology also hints at a broader trend: AI moving from isolated chat windows into the fabric of every application, reducing friction and keeping users in their native workflow. Critics will watch for privacy implications, as the pointer continuously captures visual context and speech. DeepMind says the system processes data locally where possible and only sends anonymised snippets to the cloud for model inference. Nevertheless, the rollout will likely spark discussions about consent and data handling in UI‑embedded AI.