Google Cloud Enhances Vertex AI Search for Healthcare with Multimodal AI

Counterpart Health, MEDITECH, and Suki are using Vertex AI Search for healthcare to improve clinicians' access to information and gain a more comprehensive view on patient health

SUNNYVALE, Calif., March 3, 2025 /PRNewswire/ -- Google Cloud today announced new generative AI (gen AI) capabilities in Vertex AI Search for healthcare, including a new feature called Visual Q&A that searches tables, charts, and diagrams, and the availability of Gemini 2.0 as one of the models now within the offering. Vertex AI Search for healthcare's new, multimodal search capabilities can help organizations give doctors, nurses, and others a more comprehensive view of patient health. 

"We're helping clinicians to work more efficiently with new multimodal capabilities embedded in Vertex AI Search for healthcare," said Aashima Gupta, global director of Healthcare Strategy & Solutions, Google Cloud. "Multimodal analysis processes diverse sources of patient data, like medical images and genetic information, for a more comprehensive understanding and improved decision-making."

Multimodal AI: A comprehensive view of patient health
Multimodal AI refers to technology that is capable of processing and integrating information from diverse sources – such as images, videos, and text. This capability is especially important in healthcare, where nearly 90% of healthcare data is in image form – such as x-rays, scans, or photos. To give healthcare professionals a comprehensive understanding of a patient's health, AI models can combine data from medical images, patient history, genetic information, and more. With these richer data inputs, multimodal AI can help providers in making more accurate diagnoses, designing more personalized treatment plans, and ultimately improving patient outcomes.

Vertex AI Search for healthcare goes multimodal
Vertex AI Search for healthcare is a specialized tool within Google Cloud's Vertex AI platform that helps developers build better assistive technology for searching and retrieving information from complex medical data within health records, medical documents, and other healthcare data sources. Now with Visual Q&A and Gemini 2.0, the search technology has even greater capabilities:

• Visual Q&A enables Vertex AI Search for healthcare to receive images such as tables, charts, or diagrams directly as an input – instead of taking the image and first converting it into text. For example, it could receive a diagram of a patient's feet indicating where they have an injury, alongside a checkbox from a form the patient filled out indicating if they are wearing therapeutic shoes, and any other symptoms indicated on the form. Visual Q&A can also analyze medical forms where information is entered into tables – and when this information is a relevant part of a search, it will include it in the findings for the clinician.
• Gemini 2.0 helps Vertex AI Search for healthcare deliver faster, more accurate answers. For example, Gemini 2.0 Flash outperforms 1.5 Pro on key benchmarks, at twice the speed. This performance boost accelerates the retrieval and processing of multimodal data, including visual medical content like charts, tables, and images, enabling faster, more informed decision-making.

Counterpart Health
Counterpart Health is an AI-powered physician enablement company focused on advancing value-based care through earlier identification and proactive management of chronic disease. The company's proprietary technology platform, Counterpart Assistant, delivers real-time, patient-specific insights directly within physicians' workflows, ensuring clinical decisions are guided by actionable data that lead to better patient outcomes. Counterpart Health is a subsidiary of Clover Health Investments, Corp.

"Counterpart Health is dedicated to helping clinicians deliver the highest quality of care to their patients," said David Tsay, MD, chief medical officer at Counterpart Health. "The advancements in Vertex AI Search for healthcare provide us with powerful new tools to achieve these goals. Multimodal AI capabilities will enable clinicians to integrate diverse types of clinical data sources, providing a more comprehensive understanding of their patients' health and facilitating personalized care management."

MEDITECH
Global electronic medical record company MEDITECH has developed an AI-powered search and summarization capability in its Expanse EHR to enable care teams to find the information they need using Google's familiar search function and gen AI. The new features announced today will further advance its EHR capabilities.

"MEDITECH's collaboration with Google Cloud will bring the latest advancements in multimodal AI to our Expanse EHR," said Cathy Turner, Chief Marketing and Nursing Executive, MEDITECH. "Building on our successful work with Google Cloud in developing AI-powered search and summarization, these advancements will further enhance our ability to deliver cutting-edge tools that empower clinicians and improve workflows."

Suki
Suki recently announced its Suki Assistant now includes patient summarization and clinical Q&A for clinicians, built on Google Cloud technology. These features help users quickly and easily find information needed to make care decisions, freeing up clinicians so they can focus on their patients.

"Google Cloud is continuing to push the boundaries of what's possible with multimodal AI in healthcare," said Punit Soni, founder and CEO of Suki. "The advancements in Vertex AI Search for healthcare complement Suki's mission to make healthcare technology invisible and assistive. By combining Suki's expertise in ambient AI with Google Cloud's powerful multimodal capabilities, we are equipping clinicians with instant access to the critical information they need, enabling them to deliver the best possible patient care."

About Google Cloud 
Google Cloud is the new way to the cloud, providing AI, infrastructure, developer, data, security, and collaboration tools built for today and tomorrow. Google Cloud offers a powerful, fully integrated and optimized AI stack with its own planet-scale infrastructure, custom-built chips, generative AI models and development platform, as well as AI-powered applications, to help organizations transform. Customers in more than 200 countries and territories turn to Google Cloud as their trusted technology partner.

SOURCE Google Cloud