Gemini’s new beta feature provides proactive responses based on your photos, emails, and more

Google announced on Wednesday that it’s launching a new beta feature in the Gemini app that allows the AI assistant to tailor its responses by connecting across your Google ecosystem, starting with Gmail, Photos, Search, and YouTube history.

Although Gemini could already retrieve information from these apps, it can now reason across your data to provide proactive results, such as connecting a thread in your emails to a video you watched. Google says this means Gemini understands context without being told where to look.

The tech giant notes that this beta experience, called Personal Intelligence, is off by default, as users have the option to choose if and when they want to connect their Google apps to Gemini. Of course, not everyone wants AI looking at their photos and YouTube history. If you do decide to connect your apps, Gemini will only use Personal Intelligence when it determines that doing so will be helpful, Google says.

“Personal Intelligence has two core strengths: reasoning across complex sources and retrieving specific details from, say, an email or photo to answer your question,” wrote Josh Woodward, VP, Gemini app, Google Labs, and AI Studio, in a blog post. “It often combines these, working across text, photos and video to provide uniquely tailored answers.”

Woodward shared an example of when he was standing in line at a tire shop and didn’t remember his car’s tire size. While most AI chatbots can determine a car’s tire size, Woodward says Gemini can go further by offering personalized responses. In his case, Gemini suggested all-weather tires after identifying family road trip photos in Google Photos. Woodward also said he forgot his license plate number, but Gemini was able to pull the number from a picture in Photos.

“I’ve also been getting excellent tips for books, shows, clothes and travel,” Woodward wrote. “Just this week, it’s been exceptional for planning our upcoming spring break. By analyzing our family’s interests and past trips in Gmail and Photos, it skipped the tourist traps. Instead, it suggested an overnight train journey and specific board games we could play along the way.”

Google says it has guardrails for sensitive topics, as Gemini will avoid making proactive assumptions about sensitive data like health. However, the tech giant also notes that Gemini will discuss this data if you ask it to.

Additionally, Gemini doesn’t train directly on your Gmail inbox or Google Photos library. Instead, it trains on specific prompts in Gemini and the model’s responses. In the examples above, the photos of the road trip, the license plate picture in Photos, and the emails in Gmail are not directly used to train the model. They are only referenced to generate a response, Google says.

Personal Intelligence is rolling out to Google AI Pro and AI Ultra subscribers in the U.S. Google plans to expand the feature to more countries and Gemini’s free tier. 

Google provided a list of example prompts to try, including “Help me plan my weekend in [city i.e. New York] based on things I like to do,” “Recommend some documentaries based on what I’ve been curious about,” or “Based on my delivery and grocery receipts in Gmail, Search history, and YouTube watch history, recommend 5 YouTube channels that match my cooking style or meal prep vibe.”