Data Analytics

How Honeylove boosts product quality and service efficiency with BigQuery

Thu, 02 Apr 2026 16:00:00 +0000

Building the perfect bra takes thousands of data points.

That’s why Honeylove isn’t just another intimates brand. We’re a technology company that happens to make exceptional bras, tops, shapewear, and bodysuits. Technology shapes everything we do, from how we iterate garments based on customer feedback to how we optimize sizing across those thousands of data points.

When Honeylove was born in 2018, though, our data wasn’t consolidated. We were looking at analytics in Shopify, checking email campaign performance in one platform, and reviewing ad metrics in another. We weren’t connecting the dots as effectively as we could have.

Then we fell in love with BigQuery. In this post, we’ll cover how Honeylove uses BigQuery and Gemini to unify our data, automate key business insights, and leverage AI to boost product quality and service efficiency — as well as how other organizations looking to make the most of their data can follow our approach intimately.

Transforming insights with BigQuery and Gemini

The first step was getting all our data in one place. BigQuery gave us exactly what we needed: a performant, economical, unified data platform that integrates seamlessly with the tools our team already uses within the Google ecosystem, such as Google Ads and Google Sheets . This helped eliminate manual data silos and enabled us to quickly adopt AI and ML capabilities across the business.

The real transformation came when we started leveraging BigQuery ML functions for contribution analysis. We built models to analyze the key drivers behind some of our most critical metrics: conversion rate, customer satisfaction scores, website performance, and return rates.

What’s really powerful for us is that we can feed these contribution analysis results directly into Gemini to produce accessible reports and summaries. Before implementing this approach, 10 to 15 people would spend an hour before key meetings manually reviewing dashboards, trying to drill into the data and find meaningful insights. We’ve saved hundreds of hours per year just by automating this process with Gemini.

But the impact of BigQuery and Gemini goes beyond time savings. These tools help us find patterns and insights we would’ve missed entirely. Even if you have the best marketing analysts looking over dashboards, they just wouldn’t be able to slice it in the same way these reports allow us to do.

We’ve also been able to transform forecasting inventory and demand planning, another area where manual processes previously dominated. By deploying and training BigQuery ML’s ARIMA univariate forecasting models, we’ve used high-accuracy SKU-level demand forecasts that automatically adjust for seasonality and recent changes.

These automated forecasts consistently come within 5% of what we calculate manually — a huge improvement over third-party vendors that were sometimes off by 20% to 30%. Having that additional checkpoint gives us more confidence when making critical inventory decisions.

Unlocking value and creative with multimodal embeddings

Customer service tickets can be a treasure trove of valuable feedback and information for ecommerce brands. But only if you can extract insights from them, and with Google Cloud, we can.

We leverage Gemini embedding models and BigQuery vector search to transform the unstructured text of our tickets into actionable data. We generate vector embeddings for tickets already in our data warehouse using simple SQL commands, and then use those vectors for semantic searching through retrieval-augmented generation (RAG).

This allows us to ask precise, natural-language questions, such as “What do customers love about our bras?” or “What changes would you like to see to our bodysuits?” In response, Gemini instantly identifies similar use cases, enabling us to move beyond keyword matching and quickly find the root causes of any issues, which are often nuanced. This proactively guides product improvements and enhances service efficiency. We’re saving about 30 seconds per ticket, which might not sound dramatic until you multiply it across thousands of interactions.

We’re also experimenting with multimodal embeddings for video asset search across our ad and influencer content library. It’s been fun to test queries like “find me videos with dogs” or “find me a video with a red dress” and watch it actually work. The next step is to use those embeddings to compare new creative assets with existing ones and predict performance based on our historical data.

Growth creative has traditionally been driven by gut feelings rather than numerical analysis, but we hope to change that by using our huge library of existing ad creative to inform what we test and create in the future.

Building for the future with Google Cloud

Today, Google Cloud and BigQuery are a central pillar of our company. They allow us to spend less time on manual tasks and more time on high-value work that solves real-world problems, making us very efficient as a small team.

Working with the Google Cloud team is invaluable. They’ve been a true partner, and they continue to support our roadmap. We’re leaning further into BigQuery ML functionality, moving more of our data science work into automated, always-available models rather than offline analyses.

We’re also developing internal knowledge bots using the Vertex AI RAG Engine, connected directly to our internal documents hosted on Google Drive, to provide instant answers to internal policy and process questions. Additionally, we’re experimenting with Conversational Analytics API to provide a “BI in a box” experience so our teams can ask plain-text questions and get metrics and charts without needing an analyst.

As a technology-first company, this transformation continues to have a profound impact on what we do at Honeylove. It accelerated innovation in product quality, improved operational efficiency, and ensured that our customers receive a more intelligent and consistent service experience.

What’s new with Google Data Cloud

Thu, 02 Apr 2026 16:00:00 +0000

March 30 - April 3

Google-built ODBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built ODBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your application to BigQuery.

March 23 - March 27

We showed you how you can scale your reads with Cloud SQL autoscaling read pools. This feature allows you to provision multiple read replicas that are accessible via a single read endpoint and to dynamically adjust your read capability based on real-time application needs.
Our customers are leveraging the full power of Conversational Analytics and Looker to drive major business and technical breakthroughs in the AI era. Companies like Telenor, Pet Circle, Fluent Commerce, Lighthouse Intelligence, Wego, and ROLLER are turning data into insights and actions, grounded by Looker’s semantic layer.

March 16 - March 20

We introduced an enhanced Gemini assistant in BigQuery Studio, transforming the agent from a code assistant into a fully context-aware analytics partner.

February 23 - February 27

We introduced managed and remote MCP support for Google Cloud databases, including AlloyDB, Spanner, Cloud SQL, Bigtable and Firestore, to power the next generation of agents. This announcement extends the ability for AI models to plan, build, and solve complex problems, connecting to the database tools our customers leverage daily as the backbone of their work environment.
We outlined how you can build a conversational agent in BigQuery using the Conversational Analytics API to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

February 16 - February 20

Our customers are leveraging the full power of Looker to drive major business and technical breakthroughs. Companies like Arrive, Audika, Carousell, Framebridge, GumGum, Intel, Overdose Digital, Ocean Network Express, Subskribe and Promevo are leveraging Looker’s newest AI-driven capabilities, including Conversational Analytics, to transform data to insights and actions, and empower their entire organization with a single source of truth, powered by Looker’s semantic layer.

February 2 - February 6

Join us on March 4 for our webinar, Win Your AI Strategy with Cloud SQL Enterprise Plus, to learn how to power your generative AI workloads with 3x higher performance and 99.99% availability. Register today to discover how to build a scalable, enterprise-grade foundation for your most demanding AI applications.

January 26 - January 30

We introduced Conversational Analytics in BigQuery, which allows users to analyze data using natural language. Conversational Analytics in BigQuery is an intelligent agent that generates, executes and visualizes answers grounded in your business context directly in BigQuery Studio, making data insights for data professionals more conversational.
We outlined how data products have become the foundation for AI agents, providing the context needed to make autonomous agents reliable and trusted for real business use, backed by organized business logic and semantic understanding.
We highlighted how you can supercharge data analytics workflows, and outlined Google Cloud’s AI agent offerings for data engineering, data science, and development tools, so you can integrate agentic workflows in your applications, empower your teams and speed discovery.

January 19 - January 23

We have fundamentally reimagined Firestore with pipeline operations for Enterprise edition. Experience a powerful new engine featuring over a hundred new query features, index-less queries, new index types, and observability tooling to improve query performance. Seamlessly migrate using built-in tools and leverage Firestore’s existing differentiated serverless foundation, virtually unlimited scale, and industry-leading SLA. Join a community of 600K developers to craft expressive applications that maximize the benefits of rich queryability, real-time listen queries, robust offline caching, and cutting-edge AI-assistive coding integrations.
Introducing Google Cloud SQL on MSSQLTips: We are highlighting a new technical guide published on MSSQLTips titled "Introducing Google Cloud SQL." This article serves as an essential resource for SQL Server administrators and developers exploring Google Cloud's fully managed database service. It provides a detailed overview of Cloud SQL capabilities, including high availability, security integration, and the seamless transition of on-premises SQL Server workloads to the cloud, making it an ideal resource for those planning their migration strategy.
We are excited to announce the Public Preview of Microsoft Entra ID (formerly Azure Active Directory) integration with Cloud SQL for SQL Server. Designed to tackle the challenge of identity sprawl in multi-cloud environments, this integration allows organizations to govern database access using their existing Microsoft identity infrastructure. Key benefits include centralized identity management, enhanced security features like Multi-Factor Authentication (MFA), and simplified user administration through direct group mapping. This feature is available for SQL Server 2022 and supports both public and private IP configurations.

January 12 - January 16

Google-built JDBC Driver for BigQuery is now available in Preview
We are excited to announce the launch of the new, Google-built JDBC driver for BigQuery. This new open-source driver provides a direct, high-performance connection for Java applications to BigQuery and is developed entirely in-house by Google. Download a new driver and connect your Java application to BigQuery.
Troubleshoot Airflow tasks instantly with Gemini Cloud Assist investigations: Cloud Composer just got smarter. We are excited to announce that Gemini Cloud Assist investigations are now available directly within Cloud Composer 3. Instead of manually sifting through raw logs, you can now simply click "Investigate" on a failed Airflow task. Gemini analyzes logs and task metadata to identify failure patterns—such as resource exhaustion or timeouts—and provides actionable recommendations driven by Gemini Cloud Assist to resolve the issue. This integration shifts the debugging experience from manual toil to automated root cause analysis, significantly reducing the time required to restore your pipelines. Learn more about AI-assisted troubleshooting.

How FM Logistic tackled the traveling salesman problem at warehouse scale with AlphaEvolve

Thu, 26 Mar 2026 16:00:00 +0000

The traveling salesman problem asks a deceptively simple question: What's the shortest route that visits every point exactly once? It's one of the hardest problems in computer science, and mathematicians have been working on it for nearly a century. It's also what FM Logistic's warehouse operators face every day in Poland.

The facility spans eight football fields. It holds over 17,700 picking locations. And across every shift, up to several dozen operators on ride-on electric trucks crisscross the floor collecting cartons, each one navigating dozens of storage locations per tour. Every unnecessary step adds up: in time, in wear on the fleet, and in delayed fulfillment.

FM Logistic, a global logistics provider operating in over 14 countries, had already optimized their routing once. Their existing model used a fast, cost-prioritized allocation logic built for real-time responsiveness. It worked well, but it made decisions step by step, which limited how well it could coordinate routes across the full warehouse. With dozens of operators working the same floor across shifts, even a small routing improvement would compound quickly.

So they turned to AlphaEvolve on Google Cloud.

Teaching an AI to write better algorithms

AlphaEvolve is an evolutionary coding agent that generates and refines algorithms autonomously using Gemini models. Rather than calculating a schedule from fixed rules, it works as a coding partner: writing new code, scoring it, and iterating until it finds a better solution than the one it started with.

The team didn't start from scratch. They gave AlphaEvolve a "seed" program: their existing algorithm, which made routing decisions one step at a time based on what looked best in the moment. This gave the agent a working baseline that already solved the problem, just not optimally. From there, AlphaEvolve used Gemini to generate variations of this code, introducing mutations and new logic to see if it could beat the human-designed original.

Measuring what good looks like

For AlphaEvolve to improve, it needs a way to measure how well each algorithm performs. FM Logistic designed a custom evaluation function using a representative dataset of 60 tours (over one hour of workforce data), letting the agent test thousands of generated algorithms against real-world conditions.

The evaluation scored every new piece of code on a primary goal: minimize the average travel distance per pick, while avoiding operational failures. The team built in specific penalties to steer the model away from unworkable solutions — things like exceeding forklift capacity, missing pending orders, assigning the same box twice, violating FIFO priority for older orders, or exceeding the computation time required for real-time operations.

The results

The new routing logic delivered immediate, measurable gains over the previous best baseline:

10.4% improvement in routing efficiency over the previous best solution.
15,000+ fewer kilometers of warehouse travel per year at full operational scale.

That efficiency gives FM Logistic room to handle larger order volumes with the same team and equipment, without adding headcount or expanding their fleet.

"Through our partnership with Google Cloud and the implementation of AlphaEvolve and Gemini, we further optimized our routing approach for fast-moving operations," said Rodolphe Bey, Group CIO at FM Logistic. "The 10.4% improvement was achieved on top of an already highly optimized baseline, where further gains are typically hard to come by. This translates directly to faster fulfillment, improved working conditions for our teams, and reduced wear on our fleet."

What the winning algorithm actually does

By running a series of experiments, each generating hundreds of candidate programs, AlphaEvolve developed a new algorithm that outperformed the previous best human-engineered one. The result is a set of clear, human-readable rules that warehouse teams can review and adjust as needs change.

The three core improvements:

Density-based starting points (Anchor selection): The previous system chose a starting mission based on the single location where the most missions overlapped. The new algorithm looks more broadly, identifying clusters of items that are close together and using those dense areas as "starting anchors" for building routes. Every tour begins with a highly efficient core.
Two-step filtering with distance simulation: To maintain real-time speed, the algorithm uses a two-stage process. First, a quick filter eliminates orders that do not fit the route's logic. Second, a precise distance simulation runs only on the best remaining candidates to find the most efficient path, without slowing down warehouse operations.
Flexible route building: If the algorithm can’t fill a truck efficiently around a specific starting point, it doesn’t force a bad route. It returns those orders to the main pool so they can be picked up by a better-fitting route later, improving efficiency across the entire warehouse.

What’s next

The Poland pilot (which is now running in production) demonstrated what evolutionary AI can do for complex routing at warehouse scale. FM Logistic is now exploring extensions — applying the algorithm to other high-volume e-commerce facilities, researching how AlphaEvolve might help optimize road transport for less-than-truckload shipments, and investigating AI-driven product placement inside warehouses to further cut travel distances.

^{This project was a collaboration between the FM logistic team including: Mateusz Klimowicz, Jarosław Urbański, Florent Martin and Alberto Brogio and the AI for Science team at Google Cloud including (but not limited to): Kartik Sanu, Laurynas Tamulevičius, Nicolas Stroppa, Chris Page, Gary Ng, John Semerdjian, Skandar Hannachi, Vishal Agarwal and Anant Nawalgaria as well asMariusz Czopiński from the Google account team as well and partners at Google DeepMind}

BigQuery Studio is more useful than ever, with enhanced Gemini assistant

Mon, 16 Mar 2026 16:00:00 +0000

Modern data teams dedicate a huge portion of their time to managing analytics overhead rather than just analyzing data. This includes tasks such as identifying necessary data, configuring schedules, or investigating the reasons behind a stalled job. Beyond these operational challenges, they also need an assistant that is versed in their data and has the context of their current work.

The latest Gemini-powered assistant in BigQuery Studio, available today, has new capabilities that allow you to interact with your data environment differently, transforming the agent from a code assistant into a fully context-aware analytics partner.

Here is a deep dive into the major improvements you can use right now.

1. Context-aware interoperability

The query editor tab and chat interface are now highly interoperable. The assistant is now aware of your active and open query tabs.

This means you no longer have to copy-paste code snippets or explain your context from scratch. Simply ask questions or request optimizations based on the active query tab, and the assistant intelligently understands exactly what code and resources you are referring to.

Advanced SQL generation: Beyond standard queries, the assistant can now generate advanced SQL that utilizes AI operators and federated queries, helping you unlock more complex analytical use cases with simple natural language prompts.

Fig 1.1 - Assistant is context-aware of the active tab and what “query” is being referred to

2. Intelligent resource discovery

As organizations grow, data gets scattered across different projects, datasets, and tables. Finding the specific resource you need can feel like finding a needle in a haystack.

The assistant in BigQuery Studio now features resource discovery, utilizing Dataplex Universal Catalog search to find resources across single or multiple projects. You can now search for a wide range of BigQuery resources, including datasets, tables, models, saved queries, and even scheduled queries. Now, you can:

Ask questions in plain English: You no longer need to remember exact table IDs. You can search using intent-based prompts like "Where can I find demographics such as age and location for new users?" or "Do I have any dataset named ecommerce?"
Deep dive into metadata: Once the assistant finds the right dataset, the conversation doesn't stop. Ask follow-up questions to understand the structure of the data before you even write a line of code, with.
- Visual schemas: The assistant displays table schemas and dataset details in a user-friendly UI directly within the chat window.
- Optimized queries: Ask "Is this table partitioned?" or "What’s the clustering on this table?" so that you write efficient queries from the start.
- Owner identification: Ask "Who owns this dataset?" if you need to request access.

Fig 1.2 -Assistant is able to search across projects to list datasets relevant to user prompt

Further, this feature respects your organization’s security policies: it only retrieves metadata for resources you actually have permission to view.

3. Instant job analysis and troubleshooting

We’ve all been there: a query that usually takes a few seconds is hanging. Or perhaps you received a bill that was higher than expected. Traditionally, this meant digging into information schemas or logs.

With the new job analysis capability, the assistant can now search both personal and project job history to provide insights.

Debug long-running queries: Instead of guessing why a job is stalling, simply copy the Job ID and ask: "Why is this job [Job ID] taking so long?" The agent analyzes the job's status and returns key statistics explaining the delay, such as slot contention, large row scans, or high data volume.
Root cause analysis: When a scheduled job fails, perform root cause analysis by asking, "Why did this scheduled job [Job ID] fail?" The assistant also provides recommendations on how to fix the problem.
Cost control: Audit your resource consumption by asking, "What are the 3 most expensive queries in the last 2 days?" The agent returns the right SQL needed to query the Information schema to get this information.

Fig 1.3 -Assistant can analyze jobs and provide optimization

With these advanced features within the Gemini-powered chat, the BigQuery Studio assistant is evolving into a context-aware, agentic partner that supports your entire data lifecycle. By simplifying resource discovery, automating SQL workflows, and streamlining troubleshooting, these enhancements allow you to focus on high-value insights instead of operational management.

To explore the full range of what the assistant can do and how to get started, visit our documentation page.

Cool stuff Google Cloud customers built, Feb. edition: Telco data reinvention; Golden State’s “G.O.A.T.T.”; John Lewis explores DORA

Thu, 26 Feb 2026 17:00:00 +0000

AI and cloud technology are reshaping every corner of every industry around the world. Without our customers, there would be no Google Cloud, as they are the ones building the future on our platform. In this regular round-up, we dive into some of the exciting projects redefining businesses, shaping industries, and creating new categories.

For our latest edition, we explore a new data approach for Vodafone and Fastweb; evaluating John Lewis Partnership’s developer platforms; the Golden State Warrior’s AI playbook; healthy, stable networks at Hackensack Meridian Health; and Ab Initio brings better context to data for AI.

Be sure to check back next year to see how more industry leaders and exciting startups are putting Google Cloud technologies to use. And if you haven’t already, please peruse our list of 1,001 real-world gen AI use cases from our customers.

Fastweb + Vodafone reimagined data workflows

Who: Following the acquisition of Vodafone Italy by Swisscom in 2025, these leading European telecom providers wanted to rethink how they serve customers and deliver timely, personalized experiences across mobile, broadband, and digital channels.

What they did: Both companies had already begun modernizing customer data workflows with BigQuery, but combining ecosystems exposed certain limits of the existing setup. In order to give every channel real-time access to accurate customer data, they implemented Spanner as a service and governance layer, delivering low-latency reads, horizontal scalability, high availability, and a fully managed environment with zero ops overhead. The team is also using Gemini to generate clear documentation directly from the code, which saves hours of manual work.

Why it matters: Using Spanner Graph allowed the organization to map lineage in a way that reflects how its platform actually works: which tables drive specific jobs, how transformations cascade, and where dependencies sit. Call centers now see more complete, up-to-date customer information, digital channels can rely on consistent data without custom integrations, and partners can access what they need with low latency through Apigee.

Learn from us: “Rebuilding our Customer 360 platform with Google Cloud services has already changed how Fastweb + Vodafone works. Workflow monitoring is simpler, pipelines are leaner, and real-time serving is now the norm. ” – Vincenzo Forciniti, IT AI Adoption & Platform Engineering Lead, Fastweb + Vodafone

John Lewis measures the value of its developer platform

Who: The John Lewis Partnership is a major UK retailer operating John Lewis department stores and Waitrose supermarkets. To power their digital transformation, they built the John Lewis Digital Platform (JLDP) to support dozens of product teams building high-quality software for johnlewis.com.

What they did: Moving beyond simple usage metrics, John Lewis developed a sophisticated, multi-stage approach to measuring the real value of their platform. They transitioned from initial speed-based metrics (like "Onboarding Lead Time") to a comprehensive model using DORA metrics and subjective engineer feedback via the DX platform. This included a custom "Technical Health" feature that uses small, automated jobs to monitor more than 35 health measures — such as Kubernetes best practices, security, and operational readiness — providing teams with real-time "traffic light" indicators of their service health.

Why it matters: By focusing on value rather than just activity, John Lewis ensured the platform was actually reducing friction for developers rather than just being a mandatory tool. Their automated Technical Health checks allow product teams to manage technical debt and security vulnerabilities proactively. This approach has decoupled centralized operations teams from individual services, leading to faster incident resolution (MTTR), fewer outages, and significant cost savings.

Learn from us: "Measurement is a journey, not a destination. Start by measuring something meaningful to your stakeholders, but be prepared to adapt as your platform evolves. The things that mattered when you were proving out the platform's viability are unlikely to be what are important several years later when your features are mature." – Alex Moss, Principal Platform Engineer, John Lewis Partnership

Hackensack Meridian Health de-risks network migration using VPC Flow Logs

Who: Hackensack Meridian Health is a leading not-for-profit healthcare organization and the largest hospital system in New Jersey. System reliability is a cornerstone value for HMH as they manage a vast network of hospitals, urgent care centers, and physician practices.

What they did: Preparing for a large-scale migration to a new Google Cloud network design, Hackensack Meridian Health used VPC Flow Logs and Flow Analyzer to eliminate the "black box" of hybrid traffic. By enabling logs on their Cloud Interconnect VLAN attachments, they captured granular telemetry — including source/destination IPs, ports, and protocols. They then exported this data to create a visual "who-is-talking-to-what" map. This allowed them to identify critical traffic patterns between on-premises data centers and specific Google Cloud regions, VPCs, and applications.

Why it matters: In a healthcare environment, even minor network disruptions can have major consequences. By mapping traffic proactively, Hudson Meridian Health pinpointed exactly which moments in the cutover carried the highest risk. This preparation allowed them to detect a migration issue in just three minutes and resolve it within five — a process that previously could have taken hours. Beyond migration, this level of visibility enables the organization to better manage capacity planning, cost attribution, and security compliance across their hybrid infrastructure.

Learn from us: "Getting a clear picture of our interconnect traffic always felt like a black box. Enabling VPC Flow Logs and feeding it into Flow Analyzer finally gave us the map we needed. Identifying those critical traffic flows before we changed any routes was key to de-risking the entire migration." — Randall Brokaw, Cloud Engineering Manager, Hackensack Meridian Health

The Golden State Warriors’ AI-powered back office

Who: The Golden State Warriors are one of the NBA’s most successful modern franchises. Behind their on-court wins are a specialized operations team who run what might be called organization’s "G.O.A.T.T." (Greatest of All-Time Technologies), a data and AI platform that helps drive game-time insights, trading decisions, and fan experience enhancements.

What they did: The Warriors transitioned from a "gut-feeling" culture to an "analytics-first" strategy by building an internal "digital brain" on Google Cloud. Using BigQuery and Gemini, the team now automates complex workflows that previously took hours, such as generating pre-game scouting reports. They use machine learning to run thousands of trade simulations that prioritize "team fit" over raw individual stats and employ computer vision to track the "shot quality" of every attempt in the NBA. On the business side, they built a content recommendation engine using the Discovery API to deliver personalized digital experiences to their global fan base.

Why it matters: This AI-driven approach narrows the decision tree for leadership, allowing them to focus human expertise on the most viable options. By automating the “science” of data processing, coaches and scouts have more time for the "art" of face-to-face training, planning, and player development. This integration has not only influenced on-court strategy — like the three-point revolution — but has also improved business efficiency, with employees now proactively bringing AI-driven ideas to the IT team rather than waiting for top-down mandates.

Learn from us: "You can never reach a point where either humans or machines are making all the decisions. The sweet spot is finding that middle ground where intuition and data converge on the same conclusion. Data helps us narrow our decision tree before we even start evaluating specific options." — Nick Manning, Senior Director of Consumer Products & Emerging Technology, Golden State Warriors

Ab Initio unlocks enterprise data for the agentic AI era

Who: Ab Initio is an enterprise software company specializing in high-volume data integration and governance. Their platform is trusted by large-scale organizations to manage complex data lifecycles across hybrid and multi-cloud environments.

What they did: To solve the challenge of grounding AI agents in accurate data, Ab Initio partnered with Google Cloud to integrate its data fabric with BigQuery, Dataplex Universal Catalog, and Gemini. They launched a suite of more than 500 metadata and data connectors that bridge the gap between legacy systems (like mainframes, COBOL, and SAS) and modern cloud environments. This integration provides field-level, end-to-end lineage, allowing Gemini to access well-documented, "AI-ready" data regardless of where it resides.

Why it matters: AI agents are only as effective as the data they can access. By using Ab Initio as a "neutral hub," enterprises can federate data from on-premises and multi-cloud sources into a single unified layer without moving the data itself. This provides the rich semantic context and lineage needed for Gemini to perform grounded, explainable reasoning. For businesses, this means faster transition from experimental AI to production-ready agentic workflows that are auditable, compliant, and capable of making complex, automated decisions.

Learn from us: "Agentic AI requires trusted, AI-ready data and metadata. Understanding the origin, quality, and meaning of information matters as much as the data itself. Gemini serves as a key component of the agentic layer, using this context to make decisions that are explainable and auditable." — Scott Studer, Head of Development, Ab Initio & Chai Pydimukkala, Data Governance, Sharing & Integration Product Lead, Google Cloud

PayPal's historically large data migration is the foundation for its gen AI innovation

Thu, 26 Feb 2026 17:00:00 +0000

With the dawn of the gen AI era, businesses are facing unprecedented opportunities for transformative products, demanding a strategic shift in their technology infrastructure. A few years ago, PayPal, a digital-native company serving hundreds of millions of customers, faced a significant challenge. After 25 years of success in expanding services and capabilities, we’d created complexity in our data analytics infrastructure. Some 400 petabytes of data was spread across a dozen siloed systems due to limitations of scale and acquisitions of companies like Venmo, Braintree, and others.

Our very success in growth and innovation had created complexity that threatened our next evolution.

To continue leading the next wave of innovation in financial services, we knew we had to modernize our data foundation. Today, we’re proud to share how PayPal successfully completed what’s arguably one of the largest data migrations in history, culminating with the move of our analytics to BigQuery, Google Cloud’s enterprise data warehouse. This effort marks a significant leap in creating the robust data framework we’ll need to expand and advance our business priorities and meet the ever-evolving financial needs of our customers.

This migration was essential, but the scale was daunting. In fact, by some measures, such as our now sunset Teradata system, we believe this was one of the biggest data migrations in history. Befitting of such history, we wanted to offer some insights into how we tackled this migration and what others might consider when undertaking a significant migration of their own.

Untapped potential of data

As one of the original digital payment pioneers, PayPal processes billions of transactions, and houses decades of valuable customer insights. We have a mountain of data — really a mountain range — that had developed over decades without being fully leveraged in the service of our customers and merchants.

Each acquisition and new service added valuable capabilities but also introduced new data challenges. For example, a small business owner might use PayPal for online sales and Venmo for local transactions. However, providing a unified view of their business required complex processes that were costly and slow.

The fragmentation of data limited our ability to offer personalized experiences to consumers, thereby reducing the potential to maximize the value of their money and hindering our ability to gain deeper insights from the data.

As the gen AI era dawned, our digital fragmentation was becoming more than just a technical inconvenience. With AI becoming a transformative force in financial services with huge potential ROI, we knew fragmented data would severely limit our ability to create the intelligent experiences customers have come to expect. These could run from further strengthening our industry-leading fraud detection models to providing a best-in-class commerce platform for merchants to help them succeed in the competitive global economy.

To get there, we had to get our disparate data platforms in order, first.

Legacy systems, modern ambitions

The scope was massive. We needed to consolidate multiple data platforms, including what’s believed to be the world’s largest Teradata deployment, along with Hadoop clusters, Redshift, Snowflake, and various other systems processing petabytes of transaction data. This migration also had to be executed while maintaining the uninterrupted security and reliability our customers depend on.

As a technology company, PayPal has considerable internal resources, so we first had to decide whether to tackle this challenge ourselves. We weighed the costs and benefits and decided that if we were to unify and scale our on-premise infrastructure to meet our future needs, the cost and time-to-complete would have been prohibitive. Plus, the innovations in AI were happening at a rapid pace in the cloud. To truly leverage the power of our data, we needed to be where that innovation is happening.

We assessed various data warehousing solutions and chose BigQuery due to its numerous advantages. It is a fully managed, cloud native platform with disaggregated compute and storage that can scale independently. It has powerful capabilities at the scale and performance we needed, and a familiar SQL interface meant a gentler learning curve for our developer community.

Most importantly, BigQuery’s native integrations with AI enable seamless and efficient data analytics.

The journey to unified data

After choosing Google Cloud as our data partner, we embarked on our historic data migration. This may sound hyperbolic, but when you consider the scale of PayPal’s business, the geographies across which we operate, the regulations within each, the sensitive and quite literally valuable nature of this data, the scope of the challenge starts to be clear.

With the help of partners and experts from Google Cloud Consulting, we migrated more than 300 petabytes of data and streamlined operations, decommissioning around 25% of workloads. And we managed this all while maintaining zero downtime of our business operations and with no impact to customers. Here are some key factors that contributed to our success.

Alignment: The first hurdle in achieving transformations at scale is aligning stakeholders on a shared goal. So, we made it an enterprise-wide priority.

Discovery and analysis: Detailed inventories of data, workloads and inbound/outbound data streams is crucial for defining scope, effort and forecasting budget. Establishing lineage allowed us to trace the origins and relationships of various components, thereby providing a clear and comprehensive view of the dependency graphs.

Strategy: It is crucial to establish fundamental principles for the migration process, such as deciding between lift-and-shift versus modernization, defining security principles, setting governance guardrails, and determining how consumption will be tracked.

Execution: We automated every possible task and developed live dashboards to continuously monitor the progress of migrations. FinOps was integrated through the migration process with clear visibility of consumption and performance.

Benefits from BigQuery and beyond

We’ve achieve faster insights. Queries are 2.5x to 10x faster, including complex queries used by data scientists. This unlocks real-time insights, enabling PayPal to personalize product recommendations, offers, and customer support.

We’ve built new AI foundations. Data accessible for model training is 16x fresher. Feature engineering, a crucial step in AI development, is improved by instant access to clean, governed data. This accelerates the development personalized financial guidance, and predictive analytics for both consumers and businesses.

We’ve optimized operations. By migrating to BigQuery Data infrastructure vendors were reduced from four to one, streamlining operations and reducing complexity. Data duplication between platforms was entirely eliminated.

Our new unified data platform in BigQuery has become the source for PayPal's next wave of innovation, enabling us to create more intuitive, personalized experiences across our entire ecosystem and to leverage the power of gen AI.

AI-powered innovation unleashed

Looking ahead, we're exploring how this unified data platform will enable us to deliver AI-powered experiences that weren't possible before, including:

Predictive fraud prevention that spots potential issues before they affect our customers.
Personalized financial insights that help merchants optimize their businesses.
Seamless payment experiences that adapt to each customer's preferences and patterns.
More intelligent risk assessment that could help expand financial access to underserved communities.

Agentic commerce and future possibilities we are now able to imagine.

Lessons for the AI era

While our migration may be extraordinary in its scale, we are not alone in our needs or ambitions. There are ample considerations for companies within and well beyond financial services who may be pondering their own data foundations at this time.

First off, do not underestimate how under-utilized your data may be, and how unorganized.

Making sure your data is centralized, accurate, and consistent paves the way for AI experimentation and deployment. Organizations that spend time cleaning up their data fabric will be able to bring machine learning and generative AI applications to market more quickly, and do so at scale.

Second, ensuring data is accessible to everyone within your organization, with the proper controls, unlocks so much potential. Data orchestration and enterprise search, coupled with generative AI, has the potential to break down longstanding organizational silos and speed up decision-making across your organization. It’s one of the most promising applications of AI.

The financial world will continue to evolve, driven by new technologies and changing customer expectations. PayPal’s data transformation shows how even established companies can reinvent themselves to stay ahead of this change — provided they're willing to tackle the fundamental challenges that stand in their way.

In doing so, we've not only preserved our position as a digital payments pioneer but set ourselves up to continue leading the next wave of innovation in digital commerce.

A developer's guide to production-ready AI agents

Wed, 25 Feb 2026 17:00:00 +0000

Something has shifted in the developer community over the past year. AI agents have moved from "interesting research concept" to "thing my team is actually building." The prototypes are working. The demos are impressive. And now comes the harder question: How do we ship this?

That question turns out to be a multi-part one. Agents don't behave like traditional software. They reason, act, and adapt, which means they need different approaches to testing, memory, orchestration, and security. The patterns that served us well for deterministic code don't fully translate.

To help developers work through these challenges, we've published a collection of guides covering the full agent lifecycle. These resources first appeared during Kaggle’s 5 days of AI Agents Intensive, and they’ve proven so popular and useful, we wanted to make sure a wider audience had access, as well.

These guides offer practical frameworks and code samples you can adapt to your own projects. Below, we'll walk through the key concepts — from agent architecture to production deployment — so you can decide where to dig deeper.

What is an agent?

At its core, an agent is an autonomous entity that reasons, takes action, and improves over time. The agent's brain is a large language model — a cognitive engine that understands tasks, generates responses, and makes decisions based on context. Unlike a static tool, an agent adapts as it works. It follows a recursive loop: Think, then Act, then Observe. Each cycle moves the agent forward, refining its approach as it goes.

Surrounding this core is the orchestration layer — the nervous system that manages communication and data flow. Think of it as a conductor coordinating specialized tools and external services. These include short-term memory (Session State) for immediate recall, long-term memory (Memory Service) for retaining past interactions, information retrieval (RAG), and modules for executing actions in the outside world (Tool Use). A security framework ensures the agent operates safely and within its intended boundaries. The goal of this architecture is to create an intelligent, helpful, and trustworthy assistant.

For a deeper exploration of these foundational concepts, see the full Introduction to Agents guide.

Tools and interoperability

For agents to be truly useful, they need to interact with tools, data sources, and other agents. Two emerging protocols offer standardized approaches to these connections.

Anthropic's Model Context Protocol (MCP) gives agents a standardized way to connect with external data sources and stateless tools. Instead of building custom integrations for every service, developers can use MCP's standardized interface to simplify development and improve interoperability.

Google's Agent2Agent Protocol (A2A) takes this further by enabling agents to communicate directly with each other, regardless of their underlying frameworks. Agents using A2A can discover each other's capabilities, negotiate how they'll interact, and collaborate on tasks through a secure and structured exchange of messages.

Together, these protocols create the foundation for agents that work within a broader ecosystem — connecting to tools, data, and each other. The Tools and Interoperability with MCP guide explains both protocols in detail with implementation examples.

Context engineering

If the LLM is the agent's brain, context engineering is the practice of feeding it the right information at the right time. This includes prompt design, retrieval mechanisms, tool selection, and conversation history — everything that shapes how the agent understands and responds to each request.

Context engineering transforms a generic model into a personalized assistant. It determines which memories to retrieve, which tools to offer, and how to frame each interaction. Effective context engineering creates agents that feel coherent and helpful across sessions. Without it, agents forget, repeat themselves, or miss the point entirely.

The Context Engineering guide covers context engineering frameworks and practical techniques for implementation.

Testing and evaluation

Autonomous agents require new approaches to quality assurance. When an agent makes its own decisions, success depends on sound judgment throughout the process, not just correct outputs.

Agent evaluation focuses on trajectories — the full sequence of decisions and actions an agent takes to reach a result, not just the final answer. Two agents might arrive at the same conclusion through very different paths, and understanding those paths matters. Good evaluation examines tool selection, reasoning quality, error recovery, and whether the agent asked clarifying questions when it should have.

A practical evaluation approach includes unit tests for individual components, trajectory analysis for multi-step decision sequences, and staged rollouts from sandbox to canary to production. Each stage validates different aspects of agent behavior before you expose it to more users.

For detailed evaluation frameworks and testing methodologies, see the Agent Quality guide.

Deploying agents to production

Moving from prototype to production requires infrastructure designed for agent-specific needs. Traditional deployment patterns need adaptation for systems that maintain state, use tools dynamically, and operate autonomously.

Production agents need session management to maintain context across interactions, persistent memory systems for long-term recall, tool integration with appropriate authentication and permissions, and real-time logging to trace agent decisions and actions.

Most teams deploy in stages: sandbox for internal testing, canary for limited real-world exposure, and production for full rollout. Each stage validates performance and catches issues before you expand access.

The Prototype to Production guide provides architectural guidance and code samples for building production-ready agent infrastructure.

Where to start

Your starting point depends on where you are in the journey. The Introduction to Agents guides covers foundational concepts, while Tools and Interoperability with MCP and Context Engineering address the practical challenges of building. When you're ready to validate and ship, Agent Quality and Prototype to Production will get you there.

The agents space is moving fast, but you don't have to figure it out alone. Pick the resource that matches your current challenge and start building.

Simplify your AI workflow with autonomous embedding generation in BigQuery

Thu, 19 Feb 2026 17:00:00 +0000

In the world of generative AI and Retrieval-Augmented Generation (RAG), embeddings are the "secret sauce" that allow machines and AI agents to understand the semantic meaning of data. As BigQuery extends its autonomous data-to-AI platform, embeddings unblock valuable multimodal use cases. However, for many data engineers, managing embeddings is a headache. Traditionally, users have to set up embedding generation pipelines themselves to propagate source content updates, embedding generation, and storage.

To help BigQuery users with their AI workloads, we’re introducing autonomous embedding generation. This feature allows BigQuery to automatically maintain an embedding column on a table based on a source column. No more manual pipelines, no more synchronization issues, just easy, AI-ready data.

Managing embeddings, the old way

Before autonomous generation, the process of updating your vector search database usually looked like this:

Detect new rows in your source table.
Generate embeddings via functions like AI.EMBED.
Handle rate limits and retries.
Update the destination table with the new vectors.
Monitor the progress of your embedding generations.

If your data changes frequently, keeping these vectors in sync can be a full-time job for the user/administrator. With this as the backdrop, we set out to enhance BigQuery with the following capabilities.

1. Help the user directly work with their data

We want to simplify the search experience for the user, so that they can do simple things like AI.SEARCH(TABLE mydataset.products, 'product_description', "A really fun toy"), without having to interact or understand the embeddings.

2. Automatic synchronization

BigQuery should manage embedding generation on behalf of the user and keep generated embeddings in sync with the source data.

3. Tight integration with vector indexes

BigQuery’s VECTOR_SEARCH has many users, and we want to ensure that the managed embedding was integrated into it.

The solution: autonomously generated embedding columns

We solved this by treating embeddings as a managed part of your table. Using a familiar SQL syntax, you can now define an autonomous embedding column that BigQuery manages for you.

code_block: <ListValue: [StructValue([('code', "CREATE TABLE mydataset.products (\r\n name STRING,\r\n description STRING,\r\n description_embedding STRUCT<result ARRAY<FLOAT64>, status STRING>\r\n GENERATED ALWAYS AS (AI.EMBED(\r\n description,\r\n connection_id => 'us.test_connection',\r\n endpoint => 'text-embedding-005'\r\n ))\r\n STORED OPTIONS( asynchronous = TRUE )\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a459760>)])]>

For more information, please refer to the guide.

Integration with vector index and vector search

In addition, BigQuery’s vector index and vector search are also integrated with the generated embedding column. You can directly create a vector index associated with the source data column and query your data without managing embeddings manually. BigQuery automatically applies the base table's model to generate compatible embeddings for your query.

Introducing AI.SEARCH

We also launched a new function, AI.SEARCH, to provide a simplified signature for you to get started with the data-centric search experience. AI.SEARCH automatically uses the embedding model associated with the generated embedding column from the base table, so you don’t need to interact with the embedding configuration when using AI.SEARCH or VECTOR_SEARCH.

code_block: <ListValue: [StructValue([('code', 'SELECT base.name, base.description, distance\r\nFROM AI.SEARCH(TABLE mydataset.products, \'description\', "A really fun toy");'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a459370>)])]>

Simple management

Autonomous embedding generation is in preview, ready for you to use as part of your data analytics pipelines today. We’ve also invested in a few features to help make the process simpler to manage end to end:

Embedding status metadata: You can track the progress of embedding generation by querying the percentage of non-null embeddings in your table:

code_block: <ListValue: [StructValue([('code', "SELECT\r\n COUNTIF(description_embedding IS NOT NULL\r\n AND description_embedding.status = '') * 100.0 / COUNT(*) AS percent\r\nFROM mydataset.products;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a459cd0>)])]>

While you can initiate the creation of the vector index at any time, generating an index model will only happen at a scale when performance will benefit.

Native access to Vertex AI models: By ensuring your BigQuery connection has the Vertex AI User role, embedding generation can securely "talk" to a remote state of the art Vertex AI embedding models on your behalf.
Native error monitoring: If any step in the embedding generation pipeline fails, , you can view the status of recent background jobs via INFORMATION_SCHEMA jobs view (example), Here you can find detailed error info to help you resolve the issue.

What’s next

Autonomous embedding generation represents a shift toward AI-native multimodal data foundation that’s built for processing and activation of all data types. By automating and coupling embedding generation within the data platform, we’re helping developers spend less time on plumbing and more time on building intelligent applications. And we’re not done yet, and are hard at work building:

Simpler connection creation via Data Definition Library (DDL) and Data Control Language (DCL)
The ability to add a generated embedding column to existing tables via ALTER TABLE ADD COLUMN DDL
API and UI support for managing generated embedding columns
Direct support for multimodal data using ObjectRef

Ready to try it? Check out the official BigQuery documentation to set up your first managed embedding table today.

Building a conversational agent in BigQuery using the Conversational Analytics API

Thu, 19 Feb 2026 17:00:00 +0000

Bringing data into BigQuery centralizes your information, but the real challenge is making that data accessible. Often, technical barriers separate the people with questions — from execs to analysts — from the answers they need.

With the Conversational Analytics API, powered by Gemini, you no longer need intricate systems to get insights. The API is engineered to help you build context-aware agents that can understand natural language, query your BigQuery data, and deliver answers in text, tables, and visual charts.

Now, you can build any solution that can interface with the API. For example, you can integrate it with the Agent Development Kit (ADK) to build a multi-agent systems, or to implement these data strategies:

Self-service triage for operations: Give teams like Support and Sales an agent that answers data questions instantly. Instead of filing a ticket to ask, "Why did signups drop last week?", they get the answer immediately.
Differentiate your SaaS product: Differentiate your platform by embedding a powerful chat interface directly into your platform. Let your customers query and visualize their own usage data using plain English.
Dynamic reporting: Move beyond static PDFs. Automate the core reporting function and enable stakeholders to ask nuanced, follow-up questions for deeper investigation, effectively replacing report versions with real-time conversation.

In this post, we’ll share ways to build a conversational agent in BigQuery using the Conversational Analytics API.

Step One: Configure and create the agent

The deployment of a Data Analytics Agent involves configuring its access, context, and environment before making the final creation call.

In our included example, the Python SDK is used, but the Conversational Analytics API supports many other languages, depending on your preference and environment.

Initialize the client and define BigQuery sources

Begin by instantiating the necessary client (DataAgentServiceClient) to interact with the API. This client is used in conjunction with explicit BigQueryTableReference objects, which authorize the agent's access to specific tables (defined by project_id, dataset_id, and table_id). These individual references are then aggregated into a DatasourceReferences object under the bq field.

code_block: <ListValue: [StructValue([('code', 'from google.cloud import geminidataanalytics\r\n\r\n# Set project-specific variables (client, location, project IDs)\r\ndata_agent_client = geminidataanalytics.DataAgentServiceClient()\r\nlocation = "global"\r\nbilling_project = "your-gcp-project-id"\r\ndata_agent_id = "google_trends_analytics_agent"\r\n\r\n# Define the BigQuery table sources\r\nbq_top = geminidataanalytics.BigQueryTableReference(\r\n project_id="bigquery-public-data", dataset_id="google_trends", table_id="top_terms"\r\n)\r\nbq_rising = geminidataanalytics.BigQueryTableReference(\r\n project_id="bigquery-public-data", dataset_id="google_trends", table_id="top_rising_terms"\r\n)\r\ndatasource_references = geminidataanalytics.DatasourceReferences(\r\n bq=geminidataanalytics.BigQueryTableReferences(table_references=[bq_top, bq_rising]))'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a95df40>)])]>

Set the agent context

Construct the context object by bundling the system_instruction (defining the agent's behavior/role) and the datasource_references (defining its permitted data access). This complete Context is then nested within the DataAnalyticsAgent structure of the final DataAgent object.

While you can provide a string based system instruction, we recommend that you use the more robust context object to provide instruction to the agent. The object can still be provided with additional system instructions to help provide supplemental guidance.

code_block: <ListValue: [StructValue([('code', '# Set the context using our system_instruction string\r\npublished_context = geminidataanalytics.Context(\r\n system_instruction=system_instruction,\r\n datasource_references=datasource_references\r\n example_queries=example_queries\r\n)\r\n\r\ndata_agent = geminidataanalytics.DataAgent(\r\n data_analytics_agent=geminidataanalytics.DataAnalyticsAgent(\r\n published_context=published_context\r\n ),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe5573b2130>)])]>

Create the agent

Call data_agent_client.create_data_agent. This request includes the parent resource path (projects/{billing_project}/locations/{location}), the unique data_agent_id, and the fully configured data_agent object to complete the deployment.

code_block: <ListValue: [StructValue([('code', '# Create the agent\r\ndata_agent_client.create_data_agent(request=geminidataanalytics.CreateDataAgentRequest(\r\n parent=f"projects/{billing_project}/locations/{location}",\r\n data_agent_id=data_agent_id,\r\n data_agent=data_agent,\r\n))'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c91ba30>)])]>

Your agent now exists and is defined by that published_context.

Step two: Creating a conversation (stateful vs. stateless)

The Conversational Analytics API can handle conversations in two ways:

Stateless: You send a question and the agent's context. You must manage the conversation history in your own application and send it with every new request.
Stateful: You create a "conversation" on the server. The API manages the history for you. This is what allows users to ask follow-up questions.

We'll configure a stateful conversation. We create a conversation object associated with our new agent.

code_block: <ListValue: [StructValue([('code', 'def setup_conversation(conversation_id: str):\r\n data_chat_client = geminidataanalytics.DataChatServiceClient()\r\n conversation = geminidataanalytics.Conversation(\r\n agents=[data_chat_client.data_agent_path(\r\n billing_project, location, data_agent_id)],\r\n )\r\n request = geminidataanalytics.CreateConversationRequest(\r\n parent=f"projects/{billing_project}/locations/{location}",\r\n conversation_id=conversation_id,\r\n conversation=conversation,\r\n )\r\n try:\r\n # Check if it already exists\r\n data_chat_client.get_conversation(name=data_chat_client.conversation_path(\r\n billing_project, location, conversation_id))\r\n except Exception:\r\n response = data_chat_client.create_conversation(request=request)\r\n print("Conversation created successfully.")\r\n\r\nconversation_id = "my_first_conversation"\r\nsetup_conversation(conversation_id=conversation_id)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c91b340>)])]>

Step three: Create a streaming chat loop

To allow for interactive analysis, we implement a function, stream_chat_response, to manage the conversation flow. The Data Analytics Agent API is designed to return a response as a stream, which is crucial for delivering updates on the agent’s progress in real-time.

A typical response stream can include distinct components, such as:

Schema: Confirmation of table resolution.
Data (query): The generated SQL query (excellent for debugging and transparency).
Data (result): The resulting data structure (e.g., a Pandas-like DataFrame).
Chart: A Vega-Lite JSON specification for data visualization.
Text: The final, synthesized natural language summary.

Defining the function

The function is defined to accept the user's question. Inside, we initialize the DataChatServiceClient and define a simple flag (chart_generated_flag) to track if a chart needs to be rendered after the stream completes. The user's question is wrapped in a Message object, which is required for the API request.

code_block: <ListValue: [StructValue([('code', "def stream_chat_response(question: str):\r\n data_chat_client = geminidataanalytics.DataChatServiceClient()\r\n chart_generated_flag = [False] # Flag to help with visualization\r\n \r\n # Format the user's question into an API-ready Message object\r\n messages = [\r\n geminidataanalytics.Message(\r\n user_message=geminidataanalytics.UserMessage(text=question)\r\n )\r\n ]"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe5742f5220>)])]>

Processing the stream

The ConversationReference is essential as it ties the current request to the stateful conversation and links it back to the specific data_agent we created earlier. Once the request object is fully assembled with the parent path, messages, and reference, we call data_chat_client.chat.

We then iterate over the returned stream. A utility function, show_message, is used here to parse and appropriately format the different response types (Text, Chart, Data) for the user. Finally, if the chart_generated_flag was set during the stream, a post-processing utility (preview_in_browser) handles the rendering of the visualization.

code_block: <ListValue: [StructValue([('code', '# Reference the stateful conversation and the created Data Agent\r\n conversation_reference = geminidataanalytics.ConversationReference(\r\n conversation=data_chat_client.conversation_path(\r\n billing_project, location, conversation_id\r\n ),\r\n data_agent_context=geminidataanalytics.DataAgentContext(\r\n data_agent=data_chat_client.data_agent_path(\r\n billing_project, location, data_agent_id\r\n ),\r\n ),\r\n )\r\n \r\n # Prepare the chat request\r\n request = geminidataanalytics.ChatRequest(\r\n parent=f"projects/{billing_project}/locations/{location}",\r\n messages=messages,\r\n conversation_reference=conversation_reference,\r\n )\r\n \r\n # Process the streaming response\r\n stream = data_chat_client.chat(request=request)\r\n for response in stream:\r\n # \'show_message\' is a utility function that formats\r\n # and prints the different response types (text, data, chart)\r\n show_message(response, chart_generated_flag)\r\n\r\n # If a chart was generated, \'preview_in_browser\'\r\n # is a utility to save and serve it as HTML\r\n if chart_generated_flag[0]:\r\n preview_in_browser()'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe554c68f10>)])]>

Step four: Talk to the agent

Asking questions

Now for the payoff. We can use our stream_chat_response function to have a conversation.

Checking the context

Let's start by seeing if the agent understands its own context.

Python

code_block: <ListValue: [StructValue([('code', 'question = "Hey what data do you have access to?"\r\nstream_chat_response(question=question)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe554c682e0>)])]>

The agent will respond with a summary of the top_terms and top_rising_terms tables, using the descriptions we provided in the system_instruction.

Natural language to SQL to Chart

Now for a complex query. Notice we ask for a chart in plain English.

Python

code_block: <ListValue: [StructValue([('code', 'question = "What are the top 20 most popular search terms last week in NYC based on rank? Display each term and score as a column chart"\r\nstream_chat_response(question=question)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe554c68b20>)])]>

The agent will stream its process:

It will show the SQL query it generated to hit the top_terms table, filtering by dma_name = 'New York NY' and the most recent week.
It will print the resulting data as a table.
It will generate a Vega chart specification.
The preview_in_browser utility will serve this as an index.html file, showing a column chart.

The stateful follow-up

This is where the stateful conversation (Step 2) pays off.

Python

code_block: <ListValue: [StructValue([('code', 'question = "What was the percent gain in growth for these search terms from the week before?"\r\nstream_chat_response(question=question)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe554c68370>)])]>

The agent remembers "these search terms" refers to the results from Question 2. It will generate a new query, this time INNER JOIN-ing the top_terms and top_rising_terms tables (as guided by our join_instructions) to find the percent_gain for that same list of terms.

Step five: Managing the agent

For a more in depth lifecycle management of the agent and messages, visit the Conversational Analytics API documentation page for the many various API requests you can make (HTTP / Python). You will find information on how to manage agents, how to invite new users to collaborate via the SetIAM, GetIAM APIs, and more.

Pro tip: Bridge the gap between data and people

By providing clear system instructions and schema descriptions, you can build an agent that is more than just conversational, as it becomes a domain expert. This interactive approach moves beyond static dashboards to provide truly accessible data analysis.

Get started

Unlocking enterprise data to accelerate agentic AI: How Ab Initio does it

Wed, 18 Feb 2026 17:00:00 +0000

Your AI agents are only as good as the data behind them. For enterprise teams building agentic AI, that's both the opportunity and the core question: How do you give Gemini and other AI models access to accurate, well-documented data when that data lives across dozens of systems, from modern cloud services to legacy mainframes?

It's a question many organizations are actively working through. Gemini and other AI models depend on large volumes of AI-ready data to support agentic workflows.

Most enterprises now store data in many places: different cloud providers, on-premises servers, and legacy systems. Pulling together the data and metadata necessary for effective AI agents requires connecting all of it.

To address the data challenges of the AI era, Google Cloud and Ab Initio are announcing a suite of products, including new data connectors, metadata connectors, and agents. Taken together, these can help build agentic AI experiences that enable autonomous actions and accelerated human-in-the-loop decisions. Ab Initio’s data integration, governance, active metadata, and agentic AI capabilities integrate directly with Google Cloud platform, notably BigQuery, Dataplex Universal Catalog, and Gemini.

How Google's Data Cloud powers agentic AI

Google's Data Cloud is built to deliver the data and context needed to power agentic AI. BigQuery provides data storage within Google Cloud along with scalable analytics and processing. Dataplex organizes data and AI assets and metadata across Google Cloud, acting as your catalog for AI to provide a dynamic system of record delivering discoverability and essential business context, including definitions, constraints, and relationships, required for AI to reason and act at scale.

Many organizations, though, operate in multi-cloud environments where data is distributed. Even when external data is available, it might lack the metadata that describes its origin, reliability, and business meaning. The partnership with Ab Initio overcomes hurdles like these.

A unified hub for the hybrid enterprise

Ab Initio serves as a neutral hub that creates a multicloud enterprise data fabric. In particular, Ab Initio extends Dataplex with a bi-directional metadata exchange across more than 500 sources. That range matters — it covers everything from modern cloud services to legacy mainframes.

The integration provides field-level, end-to-end lineage from over 100 extractors, including native converters for technologies that are both contemporary and that have long, often complex legacy systems, like COBOL, DataStage, Informatica, and SAS.

For Google Cloud customers, Ab Initio federates data from across on-premise and multi-cloud environments into a single unified layer, enabling agentic applications.

Here's how the components work together:

Ab Initio unifies access to data and metadata across systems while providing lineage and transformation context. This lineage history allows you to travel back in time to any point and answer questions about the state of metadata, supporting auditability and compliance.
BigQuery stores data and executes large-scale analytics and modeling, including on external distributed data. This means your analytics can span data wherever it lives.
Dataplex Universal Catalog extends its automated governance foundation to deliver trusted semantic context required to ground AI agents and accelerate data insights. By integrating with the Ab Initio Metadata Hub, you can manage metadata across the entire multi-cloud environment.
Gemini consumes comprehensive data and metadata for grounded, explainable reasoning and agentic activity. The richer the context, the better the AI can reason about your data.

Figure 1. Ab Initio as the Hub for Hybrid and Multi Cloud Data and Metadata

Governance across the full data and AI lifecycle

In this model, data remains distributed and heterogeneous while metadata becomes unified and standardized. Ab Initio's architecture is proven in production in the world's largest enterprises.

Ab Initio comprehensively spans the full data engineering lifecycle: transformation, quality, lineage, governance, and orchestration all working together. This produces richer metadata that can support accurate and explainable Gemini reasoning.

Context as the foundation for agentic AI

Agentic AI requires trusted, AI-ready data and metadata. Understanding the origin, quality, and meaning of information matters as much as the data itself. Gemini serves as a key component of the agentic layer, using this context to make decisions that are explainable and auditable.

Ab Initio's integration with BigQuery, Dataplex, and Gemini helps create that understanding for multi-cloud enterprises. By using Ab Initio as a hub, you can deploy agents that work with distributed data while maintaining transparency and control. The hub supports explainability, compliance, and operational reliability.

Get started

As you expand your data capabilities and incorporate agentic AI into your workflows, maintaining connected context across data sources becomes essential. Ab Initio provides the data and context to enable Google Gemini agents to operate effectively across hybrid and multi-cloud environments.

To explore the integration, visit Ab Initio's Google Cloud partner page or contact your Google Cloud representative.

_{Ab Initio offers an agentic data platform for large-scale data processing and governance, trusted by the world's most demanding enterprises. It combines active metadata with transparent, high-performance data integration to support AI-driven analytics and automation — helping organizations modernize complex systems, reduce risk, and deliver trusted data products faster.}

New BigQuery global queries let you explore distributed data with a single SQL statement

Tue, 17 Feb 2026 17:00:00 +0000

In a global economy, data is generated and stored all over the world. For a multinational corporation, customer data may reside close to headquarters in the US, while transaction logs are split between Europe and Asia, helping to meet performance, regulatory compliance, and data sovereignty needs. However, this creates a challenge: How do you get a single, unified view of your business if your data is spread across continents?

Historically, the answer has involved complex, time-consuming, and costly ETL pipelines to copy and centralize data before it could be analyzed. This introduces delays, adds complexity, and hinders teams from performing timely, ad-hoc analysis.

Today, we’re changing that with global queries in BigQuery, which lets you query data stored in different geographic locations with a single, standard SQL query – no ETL required. Global queries are available in preview.

What are global queries?

Global queries break down the barriers between datasets distributed across different geographies. Our multinational corporation can now join, union, and analyze all of its data together in a single query, directly from the BigQuery console.

In the background, BigQuery automatically handles the data movement required to execute the query, giving you a seamless, zero-ETL experience for multi-location analytics. BigQuery identifies different parts of the query that must be executed in different regions and runs them accordingly. Next, results of these partial queries are transferred to a selected location when the main query is run (with an optimization attempt to minimize the size of transfer). Finally all parts are combined and the whole query returns the results.

BigQuery user EssilorLuxottica is an early global query adopter, and has seen great results with the new feature:

“At EssilorLuxottica, we place the highest importance on data security and compliance. That’s why information is always stored in the region where it originates. With BigQuery’s global queries, we can seamlessly bring this distributed data together. This allows us to perform cross-region, aggregated analysis without compromising compliance. A powerful way to stay both secure and insight-driven.” - Rubens Ballabio, Customer Data Platform Manager, EssilorLuxottica

At the same time, transferring data across regions brings additional costs and various regulations often prohibit PII data leaving the original location. For these reasons we recognize that transferring data across geographic boundaries requires robust governance. Global queries are designed with security and user control as top priorities. Here’s how:

You must explicitly opt in: The global query feature is disabled by default to prevent accidental data transfers or costs. To use it, administrators must explicitly enable global queries for their projects. Additionally, running a global query requires special permission per user or service account.
You control the location: You specify the location where a global query is executed, allowing you to control where your data is processed, so you can adhere to data residency and compliance requirements.
By respecting governance controls: Global queries respect your existing security posture, including VPC Service Controls, so that data doesn’t move in a way that violates established policies.

Global queries in action

Let's look at a practical example: a consolidated report of global sales, generated with a single query. Before, this would require a data engineering project. Now, any authorized analyst can get the answer in seconds with a single query:

code_block: <ListValue: [StructValue([('code', "-- Set the location where the final query will be executed, can also be set in the Console.\r\nSET @@location = 'US';\r\n\r\n-- Combine transactions from both European and Asian datasets\r\nWITH transactions AS (\r\n SELECT customer_id, transaction_amount FROM `eu_transactions.sales_2024`\r\n UNION ALL\r\n SELECT customer_id, transaction_amount FROM `asia_transactions.sales_2024`\r\n)\r\n\r\nSELECT\r\n c.customer_name,\r\n SUM(t.transaction_amount) AS total_sales\r\nFROM\r\n hq_customers.customer_list AS c\r\n LEFT JOIN transactions AS t\r\n ON c.id = t.customer_id\r\nGROUP BY\r\n c.customer_name\r\nORDER BY\r\n total_sales DESC;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acf1550>)])]>

As you can see, this is all standard SQL: it takes all transactions stored in European and Asian datasets and then joins them with customer data kept in the US. What’s different is that BigQuery now executes it across datasets that are thousands of miles apart. This both dramatically simplifies your architecture and accelerates your time to insight.

Get started with global queries today

With global queries, you can start to unlock new insights across distributed data, and empower your teams with true, on-demand global analytics.

To learn more about how to enable and use this feature, please visit the official documentation. We are continuing to enhance this capability and look forward to your feedback as we progress towards general availability.

Introducing Conversational Analytics in BigQuery

Thu, 29 Jan 2026 17:00:00 +0000

Businesses want to move quickly and make informed decisions, but the explosion of data in today’s organizations often can leave knowledge teams buried and business users waiting in lengthy queues for the data insights they need. AI agents promise to fundamentally change this relationship, empowering users to move faster from data to action.

Today, we are unveiling Conversational Analytics in BigQuery in preview. This new offering allows users to analyze data using natural language, breaking down the knowledge walls and time sinks that have long been the norm. Following the general availability of Conversational Analytics in Looker, this integration brings a sophisticated AI-powered reasoning engine directly into BigQuery Studio.

Data insights for data professionals, made conversational

Conversational Analytics in BigQuery is more than a simple chatbot. It’s an intelligent agent that leverages the latest Gemini models to generate, execute, and visualize answers grounded in your specific business context directly in BigQuery’s secure and scalable environment.

With Conversational Analytics in BigQuery, technical and business teams can build and deploy intelligent agents at the source, leveraging their existing data and metadata for rapid innovation. Now you can create context- and business-aware agents right where the data lives, so that all users get smart insights, coached by trusted analysts, without ever having to wait for answers in a queue or learn SQL!

Fig.1 - Conversational Analytics in BigQuery

From question to trusted answer in seconds

Unlike simple data tools, Conversational Analytics in BigQuery uses your business metadata and production logic to build trust between the user and their data.

When a user asks a question, the agent employs a multi-stage workflow to guarantee that every response is both precise and contextually relevant. These AI-driven insights provide users with comprehensive analysis through summarized answers, raw data results, and visualizations, supplemented by follow-up questions to facilitate further investigation.

Fig.2 - Conversational Analytics in BigQuery

Conversational Analytics in BigQuery is characterized by:

Being grounded in reality: By leveraging your BigQuery schema, metadata, and custom instructions, the agent ensures SQL generation is based on internal logic rather than generic assumptions.
Verified queries and trusted logic: To maintain consistency with production metrics, you can ground the agent in verified queries and User Defined Functions (UDFs). This leverages your team’s enterprise-ready assets so you don’t have to reinvent the wheel.
Transparent logic and summarization: To give you confidence, the agent surfaces its "thinking process" and the generated SQL behind every answer. It then synthesizes the insights it gained across thousands of rows and provides a concise executive summary explaining the "why" behind the numbers.
Security and governance by design: Users only access data they are authorized to view, with every query logged for auditing within the BigQuery compliance framework.

Beyond queries: Predict what’s next in seconds

While most tools report on data retrospectively, Conversational Analytics in BigQuery transforms the experience from retrospective to predictive. By leveraging BigQuery AI, agents can forecast outcomes and uncover hidden patterns using simple language.

Behind the scenes, the agent uses functions like AI.FORECAST to predict trends or AI.DETECT_ANOMALIES to surface outliers in real-time. This allows any user to perform advanced predictive analytics in seconds, without leaving the chat interface. The agent leverages generative AI to distill millions of rows into a clear story, quickly making insights that are contextual and easy to share.

Fig.3 - Leveraging BigQuery AI functions for predictive analytics

Unlocking the value of unstructured data

With Conversational Analytics in BigQuery, you’re not limited to data in rows and columns. The agent can reason across unstructured data, such as images stored in BigQuery object tables. This lets you query your entire data estate from a single interface, transforming previously inaccessible information into actionable insights with no manual processing.

Fig.4 - Conversational Analytics in BigQuery supports unstructured data

Bring agents to life

We built Conversational Analytics in BigQuery to let you transform raw data into active, intelligent agents with minimal effort. By simply connecting your tables and adding specific business instructions and metadata, you can move beyond manual queries to automated insights. BigQuery's assisted authoring helps you create quality agents quickly, which can then be shared across Looker Studio Pro the BigQuery UI.

The agents can also be integrated into their own custom apps and existing agentic ecosystems via the API and ADK tools.

Transform your data analytics today

If you’re ready to tackle the data analytics bottleneck, you can access the preview of Conversational Analytics in BigQuery starting today. For more information, including a deep dive into best practices for context-based grounding and API integration, please refer to our documentation or learn more about Google Cloud’s AI agents for data analytics.

What's new with ML infrastructure for Dataflow

Tue, 27 Jan 2026 17:00:00 +0000

The world of artificial intelligence is moving at lightning speed. At Google Cloud, we’re committed to providing best-in-class infrastructure to power your AI and ML workloads. Dataflow is a critical component of Google Cloud’s AI stack that lets you create batch and streaming pipelines that support a variety of analytics and AI use cases. We’re excited to share a wave of recent features and capabilities that give you more choice, greater obtainability, and improved efficiency when it comes to running your batch and streaming ML workloads.

More choice: Performance-optimized hardware

We understand that all ML workloads are not created equal. That's why we're expanding our hardware offerings to give you the flexibility to choose the best accelerators for your specific needs.

New GPUs: We’re constantly adding the latest and greatest GPUs to our lineup, and we recently announced support for NVIDIA H100 GPUs (A3 High and A3 Mega VMs with enhanced networking capabilities). This means you can take advantage of cutting-edge hardware to accelerate your AI inference workloads. Leading businesses are leveraging GPUs in Dataflow to power innovative customer experiences — from threat intelligence platform provider Flashpoint powering document translation, to media provider Spotify enabling at-scale podcast previews.
TPUs: For large-scale ML tasks, Tensor Processing Units (TPUs) offer a powerful and cost-effective solution. We recently announced support for TPU V5E, V5P and V6E, which enable state-of-the-art ML builders to efficiently run high-volume, low-latency machine learning inference workloads at scale, directly within their Dataflow jobs.

Greater accelerator obtainability

Getting access to the hardware you need, when you need it, is crucial for keeping your ML projects on track. We've introduced new ways to consume accelerators that make it easier than ever to get the resources you need.

GPU/TPU reservations: You can now reserve GPUs and TPUs for your Dataflow jobs, so that you'll have the resources you need when you need them. This is important for critical workloads that can't afford to wait for resources to become available.
Flex-start GPU provisioning: For batch jobs with flexible start times, securing GPUs can be a manual and uncertain process due to high industry-wide demand. Our new flex-start provisioning model enabled by Dynamic Workload Scheduler (DWS) effectively addresses this issue: Instead of a job failing when accelerator resources are unavailable, Dataflow now queues your job and automatically starts it as soon as the required GPUs become available. This eliminates the need for repeated manual resubmissions, mitigating stockout risk and accelerating developer productivity.

Improved efficiency for AI workloads

We're always looking for ways to help you run your AI workloads more efficiently. Recently announced features like right fitting and ML-aware streaming are designed to help you get the most out of your investments in AI.

ML-aware streaming: We’ve made our streaming engine ML-aware, which means it can now make smarter decisions about how to execute your streaming ML pipelines. For example, horizontal autoscaling in Dataflow usually relies on a variety of input signals that include backlog size and CPU utilization. As accelerators become a critical part of the compute infrastructure for processing ML workloads, they need to be factored into autoscaling decisions. We recently launched GPU-based autoscaling, which enables the Dataflow service to use GPU-related signals such as degree of parallelism as a key input to its horizontal autoscaling algorithm, and deliver increased efficiency to streaming ML jobs.
Right fitting: Dataflow pipelines often have different resource needs at different stages. Previously, you had to choose a single machine type that would be powerful enough for the most demanding stage, leading to inefficiency and wasted cost during the less resource-intensive stages of the job. Right fitting solves this "one-size-fits-all" problem by allowing you to use heterogeneous resource pools. This means that compute-intensive stages of a job can run on specialized hardware with high memory or GPUs, while other stages run on generic, cost-effective workers. This leads to significantly increased efficiency and optimized costs.

We’re incredibly excited about Dataflow’s ML capabilities and the possibilities they unlock for our customers. Get started with Dataflow today and use these features to solve your hardest ML challenges.

BigQuery AI supports Gemini 3.0, simplified embedding generation and new similarity function

Mon, 26 Jan 2026 17:00:00 +0000

The digital landscape is flooded with unstructured data — images, videos, audio, and documents — that often remain untapped. To help you unlock this data's potential with minimal friction, we have integrated Gemini and other Vertex AI models directly into BigQuery, simplifying how you work with generative AI and embedding models using BigQuery SQL.

New launches in this area further simplify setup and expand what you can do with AI functions:

Simplified permission setup by using End User Credentials (EUC)
AI.generate() function for both text and structured data generation
AI.embed() function for embedding generation
AI.similarity() for computing semantic similarity scores between text and images
Gemini 3.0 Pro/Flash support

Streamlined setup with EUC

Previously, when integrating Vertex AI models with BigQuery, you needed to configure a separate connection and manage service account permissions. You can now authenticate Vertex AI requests using your personal IAM identity by enabling EUC. This eliminates the need for intermediary connections for standard interactive queries, making the connection_id parameter optional. To utilize EUC, simply ensure your account has the Vertex AI User role granted in IAM. See the screenshot below which illustrates the steps, or our public doc for more details. Note that if you are a project owner, you don’t even need to do this setup as you have the permission already.

Next gen text and structured generation functions in GA

The next generation of BigQuery gen AI functions — AI.GENERATE and AI.GENERATE_TABLE, previously in preview, are now in GA. With these new functions, BigQuery's generative AI inference capabilities let you:

Analyze any type of data: The new functions accept any type of input — text, images, video, audios and documents.
Accomplish most major AI/ML tasks: Provide the prompt of what you desire the LLM to do and perform extraction, translation, summarization, sentiment analysis etc. tasks with ease.
Use AI anywhere in your SQL: These functions are fully composable and can be placed anywhere standard SQL functions can go: in the SELECT statement, WHERE clause, and ORDER BY clause, allowing for sophisticated and flexible data processing.
Generate structured output: Convert your unstructured data to structured insights by specifying your desired output_schema.

AI.GENERATE is excellent for free-form text generation, which is useful for a wide range of generic LLM tasks such as summarization, translation, sentiment analysis, and more, all from a simple user prompt.

Additionally, AI.GENERATE can also generate structured output. By using the output_schema parameter, you can define the names and types of output fields, making the results immediately parseable and ready for use in downstream applications.

Furthermore, by specifying descriptive output field names like "sentiment" or "summarize_in_one_sentence", AI.GENERATE can accomplish multiple AI tasks with a single function call, returning the results in multiple, easily consumable columns.

We illustrate this below using two examples. The first example uses text data in the bigquery-public-data.bbc_news.fulltext table. A single AI.GENERATE call simultaneously performs five tasks: 1) key entity extraction; 2) topic modeling; 3) sentiment analysis; 4) translation; and 5) summarization.

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n title,\r\n body,\r\n AI.GENERATE(\r\n body,\r\n output_schema =>\r\n "key_entities ARRAY<STRING>, main_topics ARRAY<STRING>, sentiment STRING, translate_to_chinese STRING, summary_one_sentence STRING").*\r\n EXCEPT (full_response, status)\r\nFROM bigquery-public-data.bbc_news.fulltext\r\nWHERE category = \'tech\'\r\nLIMIT 3;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57d1d7550>)])]>

Running the above query gives the following output:

The second example involves analyzing the images. First, create a BigQuery external table that points to the images in Cloud Storage.

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE EXTERNAL TABLE bqml_tutorial.product_images\r\n WITH CONNECTION\r\n DEFAULT OPTIONS (\r\n object_metadata = 'SIMPLE',\r\n uris = ['gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*.png']);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57d1d74f0>)])]>

Next, run the following query, which uses a single AI.GENERATE function call to generate the image description and extract key entities.

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n uri,\r\n STRING(OBJ.GET_ACCESS_URL(ref,\'r\').access_urls.read_url) AS signed_url,\r\n AI.GENERATE(\r\n ("What is this: ", OBJ.GET_ACCESS_URL(ref, \'r\')),\r\n output_schema =>\r\n "image_description STRING, entities_in_the_image ARRAY<STRING>").*\r\nFROM bqml_tutorial.product_images\r\nLIMIT 3'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57d1d7730>)])]>

This gives you the following results. BigQuery can automatically visualize the image using its signed URL.

BigQuery also offers the AI.GENERATE_TABLE TVF, which has similar functionality as AI.GENERATE for structured output capabilities. Learn more in the official documentation and a previous blog post: Convert AI-generated unstructured data to a BigQuery table.

New simplified functions for embedding generation and computing similarity

The AI.EMBED function translates complex data into embeddings — numerical vectors where semantic similarity is represented by mathematical closeness. By converting data with AI.EMBED, you can turn abstract concepts into measurable distances, allowing you to mathematically compare items to find the best matches. Both of these features are currently available in preview.

Returning to the BBC news dataset used above, we can generate embeddings for the entire table using the following query:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n title,\r\n body,\r\n AI.EMBED(\r\n body,\r\n endpoint => "text-embedding-005"\r\n ).result\r\nFROM\r\n `bigquery-public-data.bbc_news.fulltext`\r\nLIMIT 3;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe5742dc8e0>)])]>

The following screenshot shows the output produced:

In addition, the new AI.SIMILARITY scalar function computes the semantic similarity of two pieces of text, two images, or across text and images. Under the hood, the function computes the embeddings of the two inputs and then computes their cosine similarity. To use this, imagine you want to find articles about downward trends in the housing market. You can use the following query to get the top five articles in the dataset with the most similar content:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n "housing market downward trends" AS query,\r\n title AS bbc_news_title,\r\n body AS bbc_news_body,\r\n AI.SIMILARITY(\r\n "housing market downward trends", body, endpoint => "text-embedding-005")\r\n AS similarity_score\r\nFROM `bigquery-public-data.bbc_news.fulltext`\r\nORDER BY similarity_score DESC\r\nLIMIT 5;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57d44c460>)])]>

The output is shown below.

This demonstrates how AI.SIMILARITY transcends simple substring searches by understanding the underlying concept of a query. It is the most streamlined way to perform semantic search in BigQuery, as it handles both embedding generation and similarity calculations in a single, elegant step — no pre-computation or complex pipeline required. This makes it an ideal choice for interactive analysis, prototyping, or joining small to medium-sized datasets where agility is key.

For use cases where you need to scale these semantic capabilities across millions or billions of rows, you can seamlessly transition to the VECTOR_SEARCH function to leverage precomputed embeddings and vector indexing.

Gemini 3.0 support

BQML supports Gemini 3.0 for its generative AI functions such as AI.GENERATE. You can invoke Gemini 3.0 using the following query.

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n body,\r\n AI.GENERATE(\r\n CONCAT("Translate into French ", body),\r\nendpoint => \'https://aiplatform.googleapis.com/v1/projects/{YOUR_PROJECT}/locations/global/publishers/google/models/gemini-3-flash-preview\',).result AS translation\r\nFROM\r\n `bigquery-public-data.bbc_news.fulltext`\r\nWHERE\r\n category = \'tech\'\r\nLIMIT 3;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c8d4700>)])]>

During preview of Gemini 3.0, you will need to specify the whole http endpoint string as in the example above. In the near future the endpoint name to be specified will simplify to endpoint => 'gemini-3-flash'.

Get started

Ready to explore your data with BigQuery’s AI functions? To get started, check out the documentation. Have feedback on these new features or have additional feature requests? Let us know at bqml-feedback@google.com.

Improving workflow orchestration with Apache Airflow 3.1 in Cloud Composer

Fri, 23 Jan 2026 17:00:00 +0000

In a world of fragmented data stacks, you need workflow orchestration that is innovative, portable, and extensible. Cloud Composer, our fully managed data and AI/ML workflow orchestration service, relies on Apache Airflow to provide a universal control plane. And today, we’re excited to share that Cloud Composer now supports Airflow 3.1 in preview.

Released in November, this update marks a significant milestone for the platform, and the first time a hyperscaler is offering Airflow 3.1.

Built on the foundation of Airflow 3

The new features in Airflow 3.1 are built on top of the revolutionary architecture introduced in Airflow 3.0:

Decoupled architecture: Robust separation between the scheduler and execution layer for better scalability and security
DAG versioning: Native support for automated DAG versioning, retaining the historical structure and run history even after you remove tasks or change logic
Powerful managed backfills: A redesigned backfill system that is now a first-class citizen, fully managed by the scheduler
Event-driven scheduling and data assets: Enhanced capabilities for triggering workflows based on assets as well as external events, like messages arriving in a message queue
and many more…

For a deeper dive into the architectural shifts of Airflow 3, you can read our previous announcement: Next Gen Data Pipelines: Airflow 3 Arrives on Google Cloud Composer.

What’s new in Airflow 3.1 on Composer?

We’ve invested heavily to ensure the new capabilities in Airflow 3.1 are robust and reliable on Google Cloud. Specifically, Airflow 3.1 introduces enhancements designed to increase oversight, improve reliability, and support global teams:

1. Human-in-the-Loop (HITL) workflows

As AI agents and automated pipelines become more complex, the need for human oversight increases. Airflow 3.1 introduces powerful Human-in-the-Loop (HITL) functionality that allows workflows to pause execution in a deferred state and wait for a person to make a decision via the Airflow UI or API call.

HITL puts you in control, whether you are approving a deployment, reviewing a generative AI model's output, or providing feedback to steer a pipeline.

To make this process even smoother, HITL operators integrate natively with Airflow Notifiers. You can configure a notifier to automatically send a message via Slack, email, or PagerDuty the moment a task pauses for input. Using these helper methods, the notification can include a direct link to the specific approval page, allowing stakeholders to respond immediately without searching through the UI.

2. Deadline Alerts

Managing Airflow Task SLAs has evolved. Airflow 3.1 replaces legacy mechanisms with smarter, proactive Deadline Alerts. You can now set specific time-based thresholds for your Dags and tasks relative to a start time or logical date.

If a critical pipeline such as a long-running ML training job exceeds its expected duration, Airflow will proactively notify you via standard notifiers. This ensures you can identify potential delays before they impact your downstream goals.

3. Native multi-language support (Internationalization)

Many data engineering teams are global, and team members speak different languages. Airflow 3.1 brings a fully localized experience to the modern React-based UI, supporting 17 languages including Spanish, French, Polish, German, Chinese (Simplified & Traditional), Japanese, and Portuguese.

4. Modern extensibility and developer experience

Beyond the core orchestration capabilities, Airflow 3.1 introduces multiple architectural improvements designed to enhance the developer experience. These updates empower teams to extend the platform's UI and build more responsive, synchronous applications.

React plugin system: Embed custom dashboards and views directly into the UI.
Inference execution: A new streaming API endpoint for watching synchronous DAGs until completion.

Why open orchestration wins over walled gardens

Some orchestration platforms attempt to consolidate everything into a single "walled garden" with ingestion and orchestration wrapped into a proprietary, opaque utility. But proprietary tools can suffer from feature lag as you wait for the vendor to address new use cases. Airflow has emerged as the industry standard for orchestration, so that you can be at the forefront of the market.

At Google, we are deeply committed to fostering this ecosystem — not just as a platform provider, but as active contributors to the Airflow codebase. Cloud Composer 3 represents the maturation of managed open source, balancing the security of a hardened perimeter with the near-infinite extensibility of open standards. With this approach, you get:

Community-led innovation: Feature velocity shouldn't be limited by a single corporation's R&D budget. With Airflow 3.1, you leverage the collective innovation of thousands of global contributors, providing fast access to a vast ecosystem of community-built providers.
Freedom from vendor lock-in: Unlike proprietary platforms where your logic is tied to a specific vendor's ecosystem, Cloud Composer runs on standard Apache Airflow. Your orchestration code remains portable Python.
Extensibility: Enterprises often manage a "long tail" of legacy integrations and niche tools. Instead of waiting for a vendor to build a connector, Composer allows you to write custom Python operators to interface with any system — from proprietary internal APIs to on-prem hardware — giving you strong connectivity without the "roadmap blockage" typical of closed tools.

Get started today

The features listed above — from HITL workflows to proactive Deadline Alerts — are a testament to the innovation behind Apache Airflow. Cloud Composer 3 with Airflow 3.1 is available now in preview. We invite you to create a new environment and explore these new capabilities today.

How Palo Alto Networks built a multi tenant scalable Unified Data Platform

Fri, 16 Jan 2026 17:00:00 +0000

Enterprises across the world are processing significant amounts of data. Palo Alto Networks processes thousands of firewall logs, telemetry signals and threat events every second across its product portfolio. To support this scale, Palo Alto Networks had 30,000 individual data pipelines, each with its own operational load. And while this single tenant architecture model worked originally, it had recently started to slow innovation, limit further scale, and made onboarding new analytics use cases increasingly costly.

To support the next generation of security products, Palo Alto Networks partnered with Google Cloud to modernize their data processing landscape into a unified multi-tenant platform powered by Dataflow, Pub/Sub and BigQuery. This transformation became the foundation of Palo Alto Network's Unified Data Platform (UDP), which now processes billions of events every day with improved agility, simpler operations and meaningful cost efficiency.

The challenge: A single tenant architecture could not keep pace

Before migrating, Palo Alto Network’s data platform was built around a “one pipeline per tenant” model. Each tenant pipeline required its own configuration, troubleshooting, on-call rotations and capacity tuning. As Palo Alto Networks usage grew, so did the friction:

Brittle alerting and weekly operational overhead to support more than 30,000 pipelines that were processing a combined throughput of roughly 30 GB per second.
Slow deployment cycles made onboarding new tenants harder.
Significant compute resources were dedicated to each tenant, regardless of load.
Engineering time was spent managing infrastructure instead of building new analytics.

This model hindered operational agility and made it challenging to scale,as new product lines expanded and data volumes increased.

The transformation: Embracing a new architectural paradigm with Dataflow

The turning point came when the team recognized that Google Cloud Dataflow’s serverless auto scaling architecture could support a completely different operating model. Instead of maintaining thousands of individual pipelines, Palo Alto Networks could unify workloads into a multi-tenant system where resources are shared intelligently across tenants.

Several core capabilities made this possible:

1. The architectural shift
Dataflow allowed the team to move from “one job per tenant” to a “shared resource pool” that can handle multiple tenants within a single architecture. This shift dramatically simplified operations and unlocked new efficiencies.

2. Unlocking multi tenancy at scale
Dataflow’s autoscaling engine manages fluctuating workloads with ease, accommodating the unpredictable spikes that are common in cybersecurity environments. This eliminated the need for manual capacity planning.

3. Operational freedom
By using Flex Templates and Dataflow’s managed service model, the team transformed their CI/CD process from week-long deployment cycles into a single day workflow. Engineers no longer spend time managing infrastructure and can instead focus on analytics, threat detection and product innovation.

4. Unified execution
With all jobs running on a shared Dataflow based platform, the team gained flexibility to move workloads across real time and batch systems without maintaining different codebases.

5. Observability
With Dataflow, the team relies on built in logging and metrics to monitor pipeline health across both real time and batch workloads, providing clear visibility into performance without additional tooling. Dataflow exposes the full set of metrics required for on-call alerting, eliminating the need to build or maintain custom metrics in the PANW codebase. When alerts trigger, the Dataflow UI enables engineers to quickly identify performance bottlenecks and take corrective action.

Architecture overview

Unified Dataflow based real time pipeline powering Palo Alto Networks UDP

The impact: A meaningful shift in value, cost and engineering focus

The migration to Dataflow did not just modernize the old system. It fundamentally changed how the engineering teams work, delivering impact across several dimensions.

The economic win: By consolidating pipelines and relying on Dataflow’s autoscaling, Palo Alto Networks achieved around 30 percent compute cost savings. These savings were driven by a reduction in redundant pipelines, better utilization of shared resources and elimination of manual capacity tuning.
The platform win: The Unified Data Platform became the long term standard for real time data processing across the company. It provides a “Dataflow native” blueprint that is scalable, repeatable and ready to support new product lines without duplicating engineering effort.
The people win: With Dataflow handling operational complexity, engineers now focus on building new analytics features instead of managing infrastructure. This shift improved morale, accelerated delivery cycles and reduced alert fatigue.

“The real differentiator for us was Dataflow’s ability to handle true multi-tenancy at massive scale. Its autoscaling engine is sophisticated enough to manage resources across thousands of tenants in a single job, which was key to unlocking around 30 percent cost savings. We moved from a world of managing more than 30,000 jobs to just a handful & that has fundamentally changed how our team operates.” — Palo Alto Networks Engineering Team

Extending the model: Use cases beyond cybersecurity

The architectural pattern Palo Alto Networks adopted is broadly applicable to any organization dealing with multi-tenant real time data at scale. Examples include:

E-commerce: powering real time dashboards for thousands of merchants on a single marketplace
Gaming: processing telemetry signals from millions of players to update leaderboards and detect fraud
Fintech: monitoring transactions from hundreds of banks to flag suspicious behavior in real time
IoT and logistics: analyzing data from fleets of vehicles to optimize routing and maintenance schedules

The same principles of multi-tenancy, shared execution and autoscaling can accelerate efficiency across many industries.

Building a sustainable data future

By standardizing on Dataflow, Palo Alto Networks has laid the foundation for long term agility in their security analytics platform. The Unified Data Platform now serves as the cornerstone of their real time data strategy, helping them innovate faster and operate with greater economic efficiency.

Their journey highlights how a flexible high performance data processing engine like Dataflow can give enterprises the confidence to scale without increasing operational overhead. It also provides a reusable playbook for teams that want to modernize their real time architectures using Google Cloud.

To learn more about how you can modernize your data pipelines, visit the Dataflow product page.

Introducing BigQuery managed and SQL-native inference for open models

Thu, 15 Jan 2026 17:00:00 +0000

BigQuery provides access to a variety of LLMs for text and embedding generation, including Google's Gemini models, Google-managed models from partners like Anthropic and Mistral. Using Gemini models and Google-managed partner models in BigQuery is simple — just create the model with the foundation model name and run inference directly in SQL queries. Today, we are bringing this same simplicity and power to any model you may choose from Hugging Face or Vertex AI Model Garden.

A SQL-native workflow with automated management

With the launch of managed third-party generative AI inference in BigQuery (Preview), you can now run open models using just two SQL statements.

This new capability delivers four key benefits:

Simplified deployment: Deploy open models using a single CREATE MODEL SQL statement with the model id string (e.g., google/gemma-3-1b-it). BigQuery automatically provisions the compute resources with default configurations.
Automated resource management: BigQuery automatically releases idle compute resources, preventing unintended costs. You can configure idle time via endpoint_idle_ttl.
Granular resource control: You can customize backend computing resources (like machine types and min/max replicas) directly within your CREATE MODEL statement to meet your performance and cost needs.
Unified SQL interface: The entire workflow — from model creation and inference to cost management and cleanup — is managed directly in BigQuery using SQL.

How it works: A practical example

Let’s take a look at the process of creating and utilizing an open model.

Step 1: Create a BigQuery managed open model
To use an open model from Hugging Face or Vertex AI Model Garden, use a CREATE MODEL statement along with the open model ID. It typically takes a few minutes for the query to complete, depending on the model size and machine types.

Hugging Face models
Specify the option hugging_face_model_id in the format of provider_name/model_name. For example, sentence-transformers/all-MiniLM-L6-v2 .

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE MODEL my_dataset.managed_embedding_model\r\nREMOTE WITH CONNECTION DEFAULT\r\nOPTIONS (\r\n hugging_face_model_id = 'sentence-transformers/all-MiniLM-L6-v2'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c8f3760>)])]>

Vertex AI Model Garden models
Specify the option model_garden_model_name in the format publishers/publisher/models/model_name@model_version. For example, publishers/google/models/gemma3@gemma-3-1b-it .

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE MODEL my_dataset.managed_text_model\r\nREMOTE WITH CONNECTION DEFAULT\r\nOPTIONS (\r\n model_garden_model_name = 'publishers/google/models/gemma3@gemma-3-1b-it'\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c8f3a90>)])]>

For demanding workloads, you can customize deployment settings (machine types, replica counts, endpoint idle time) to improve scalability and manage costs. You can also use Compute Engine reservations to secure GPU instances for consistent performance. See CREATE MODEL syntax for all the options.

Step 2: Run batch inference
Once the above CREATE MODEL job finishes, you can use it with AI.GENERATE_TEXT (for LLM inference) or AI.GENERATE_EMBEDDING (for embedding generation) on your data in BigQuery.

code_block: <ListValue: [StructValue([('code', "-- For embedding generation\r\nSELECT *\r\nFROM\r\n AI.GENERATE_EMBEDDING(\r\n MODEL my_dataset.managed_embedding_model,\r\n (\r\n SELECT text AS content\r\n FROM bigquery-public-data.hacker_news.full\r\n WHERE text != ''\r\n LIMIT 10\r\n ));\r\n\r\n-- For LLM inference\r\nSELECT *\r\nFROM\r\n AI.GENERATE_TEXT(\r\n MODEL my_dataset.managed_text_model,\r\n (\r\n SELECT 'Summarize the text: ' || text AS prompt\r\n FROM bigquery-public-data.hacker_news.full\r\n WHERE text != ''\r\n LIMIT 10\r\n ));"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c8f37f0>)])]>

Vertex AI endpoint lifecycle management and cost control
BigQuery offers flexible controls over Vertex AI endpoint lifecycle and costs through both automated and manual options.

Automated control: The endpoint_idle_ttl option enables automated resource recycling. If the model isn't used for the specified duration (e.g., INTERVAL 10 HOUR), BigQuery automatically “undeploys” the Vertex AI endpoint for you, stopping all costs.
Manual control: You can also manually “undeploy“ an endpoint to immediately stop the cost, or redeploy an endpoint using a simple ALTER MODEL statement.

code_block: <ListValue: [StructValue([('code', '-- Manually undeploy the model to save costs\r\nALTER MODEL my_dataset.managed_embedding_model\r\nSET OPTIONS(deploy_model = FALSE);\r\n\r\n-- Manually redeploy the model for the next inference job.\r\nALTER MODEL my_dataset.managed_embedding_model\r\nSET OPTIONS(deploy_model = TRUE);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c8f3340>)])]>

Easy resource cleanup
When you are done with using a model, simply drop it. BigQuery automatically cleans up all associated Vertex AI resources (like the endpoint and model) for you, so you are no longer charged for them.

code_block: <ListValue: [StructValue([('code', '-- Model deletion and all backend resource cleanup\r\nDROP MODEL my_dataset.managed_embedding_model;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acb1910>)])]>

Get started today

BigQuery’s new managed inference capability for 3P models fundamentally changes how data teams access and use third-party gen AI models. By consolidating the entire model lifecycle management into a familiar SQL interface, we're removing the operational friction and making powerful open models accessible to every BigQuery user, from data analysts to AI/ML engineers. For comprehensive documentation and tutorials, please refer to the following resources:

Read the documentation: Creating automatically-deployed open models
Try the text generation tutorial: Generate text with the Gemma model
Try the embedding generation tutorial: Generate text embeddings with open models

We look forward to seeing what you build!

Vibe querying: Write SQL queries faster with Comments to SQL in BigQuery

Wed, 14 Jan 2026 17:00:00 +0000

Crafting complex SQL queries can be challenging. Often, engineers simply want to express their data needs in plain English directly within their SQL workflow. Recently, we have seen how "vibe coding" — using natural language AI prompts to generate code — makes developing easier for everyone. That’s why we’re introducing Comments to SQL in BigQuery. This feature makes writing queries using natural language – ‘vibe querying’ – a reality.

Go from plain English to SQL code

Comments to SQL is an AI-powered feature that bridges the gap between human language and structured data queries. It helps you embed natural language expressions directly within your SQL statements, which the system then translates into executable SQL code. By automating this translation, you can write complex queries faster and spend less time writing boilerplate code.

For example, let’s say you need to calculate the business days between two dates, including weekends. With this feature, you won’t need to look up the exact functions to calculate business days between two dates. Now AI can generate the SQL date function for the natural language expression: “ How many business days are there between January 1st and March 15th, excluding weekends?” This minimizes the toil of manual SQL construction, which lets you focus on finding answers in your data.

Key functionality:

Embed natural language: You can integrate natural language expressions into your SQL queries by enclosing them within comments. For example: /* average trip distance by day of week */.
Contextual understanding: BigQuery’s AI analyzes the surrounding SQL context to accurately interpret the comments. This ensures the generated SQL aligns with your intent.
Flexible clauses: You can use natural language expressions within various SQL clauses. NL expressions can be used within various SQL clauses, including SELECT, FROM, WHERE, ORDER BY, and GROUP BY.
Complex queries: You use multiple expressions within a single SQL statement to build complex queries. For instance, you could use SELECT /* average trip distance, total fare */ FROM /* NYC taxi ride public data of 2020 */ WHERE /* day of week is Saturday */ GROUP BY /* pickup location */.
Accessible for everyone: This feature helps you perform data analysis even if you are not an SQL expert.
Refine as you go: After the initial SQL is generated, you can refine your natural language expressions and immediately see how the SQL output changes.

Helping all SQL users move faster

We want to help developers be more productive and simplify data exploration. This feature works for a wide range of users, from SQL beginners to seasoned SQL experts. Whether you’re a data analyst, software developer, business analyst, Comments to SQL helps you interact with BigQuery data more effectively. For example, SQL beginners can:

Generate summary statistics: SELECT /* average sales per region */ FROM /* sales_table */ GROUP BY /* region */.
Filter data based on criteria: SELECT * FROM /* customer_table */ WHERE /* age is greater than 30 and city is New York */.
Order results: SELECT * FROM /* product_table */ ORDER BY /* price in descending order */.

For advanced SQL users, here are some more advanced use cases:

1. Time series analysis with conditional aggregations. Handle time-series aggregation, conditional counting, and date extraction within a single query.

NL expression: SELECT /* daily average temperature, and count of days where temperature exceeded 30 degrees Celsius */

FROM /* weather_data */

WHERE /* year is 2023 */

GROUP BY /* day */

ORDER BY /* day */.

Generated SQL:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n DATE(timestamp) AS day,\r\n AVG(temperature) AS daily_avg_temperature,\r\n COUNT(CASE WHEN temperature > 30 THEN 1 ELSE NULL END) AS hot_days_count\r\nFROM\r\n `weather_data`\r\nWHERE\r\n EXTRACT(YEAR FROM timestamp) = 2023\r\nGROUP BY\r\n day\r\nORDER BY\r\n day;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a948280>)])]>

2. Multi-table joins and complex filtering: How to handle multi-table joins, date range filtering, and string-based filtering, combined with ordering.

NL expression: SELECT /* customer name, order total, and product category */

FROM /* customers */ JOIN /* orders */ ON /* customer ID */ JOIN /* products */ ON /* product ID */

WHERE /* order date is in the last month and customer region is 'Europe'*/

ORDER BY /* order total descending */.

Generated SQL:

code_block: <ListValue: [StructValue([('code', "SELECT\r\n c.customer_name,\r\n o.order_total,\r\n p.product_category\r\nFROM\r\n `customers` c\r\nJOIN\r\n `orders` o ON c.customer_id = o.customer_id\r\nJOIN\r\n `products` p ON o.product_id = p.product_id\r\nWHERE\r\n o.order_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 1 MONTH)\r\n AND c.region = 'Europe'\r\nORDER BY\r\n o.order_total DESC;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a948b80>)])]>

3. Window functions for ranking and moving averages: Handle window functions for ranking, which are typically complex to write manually. You can also group data by date parts.

NL expression: SELECT /* product name, monthly sales, and rank of products by sales within each category */

FROM /* sales_data */

WHERE /* year is 2023 */

WINDOW /* partition by category order by monthly sales descending */.Generated SQL:

code_block: <ListValue: [StructValue([('code', 'SELECT\r\n product_name,\r\n SUM(monthly_sales) AS total_monthly_sales,\r\n RANK() OVER (PARTITION BY category ORDER BY SUM(monthly_sales) DESC) AS sales_rank\r\nFROM\r\n `sales_data`\r\nWHERE\r\n EXTRACT(YEAR FROM sale_date) = 2023\r\nGROUP BY\r\n product_name, category, EXTRACT(MONTH FROM sale_date)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a948970>)])]>

4. Cohort analysis with date aggregations and user segmentation. Cohort analysis is a common but complex data analysis technique. It involves date truncation, grouping, and pivoting to display user retention over time.

NL expression: SELECT /* cohort month, count of active users in each subsequent month */

FROM /* user_activity */

WHERE /* user signup date is between '2022-01-01' and '2022-12-31' */

GROUP BY /* cohort month, activity month */ PIVOT /* activity month */.

Generated SQL:

code_block: <ListValue: [StructValue([('code', "WITH CohortData AS (\r\n SELECT\r\n DATE_TRUNC(signup_date, MONTH) AS cohort_month,\r\n DATE_TRUNC(activity_date, MONTH) AS activity_month\r\n FROM\r\n `user_activity`\r\n WHERE\r\n signup_date BETWEEN '2022-01-01' AND '2022-12-31'\r\n)\r\nSELECT\r\n cohort_month,\r\n COUNT(DISTINCT user_id)\r\nFROM\r\n CohortData\r\nGROUP BY\r\n cohort_month, activity_month\r\nPIVOT(\r\n COUNT(DISTINCT user_id)\r\n FOR activity_month IN (SELECT DISTINCT activity_month FROM CohortData ORDER BY activity_month)\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a948ca0>)])]>

Getting started

My team and I are already seeing the potential of BigQuery's NL expressions in comments to generate SQL to streamline our customers’ workflows. We're confident you'll find it a valuable addition to your BigQuery toolkit. To get started:

Open BQ Studio.
Ensure the SQL Generation Widget is enabled.

Examples:

code_block: <ListValue: [StructValue([('code', 'SELECT /* tip and passenger count columns */ FROM /* NYC taxi ride public data */ WHERE /* passenger count greater than 6 and tip is zero */'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57a9482b0>)])]>

3. Select the SQL with your comments to transform. Click on the Gemini gutter button & click the “Convert comments to SQL” button.

4. Generation widget will appear & provide a diff view of the converted SQL/NL expression.

5. Select Insert or continue to refine using the refine/multiturn feature.

Build data analytics agents faster with BigQuery’s fully managed, remote MCP server

Wed, 07 Jan 2026 17:00:00 +0000

Connecting AI agents to your enterprise data shouldn't require complex custom integrations or weeks of development. With the release of fully managed, remote Model Context Protocol (MCP) servers for Google services last month, you can now use BigQuery MCP server to give your AI agents a direct, secure, way to analyze data. This fully managed MCP server removes management overhead, enabling you to focus on developing intelligent agents.

MCP server support for BigQuery is also available via the open source MCP Toolbox for Databases, designed for those seeking more flexibility and control over the servers. In this blog post, we discuss and demonstrate the integrations of newly released fully managed, remote BigQuery Server, which is in preview as of January 2026.

Remote MCP servers run on the service's infrastructure and offer an HTTP endpoint to AI applications. This enables communication between the AI MCP client and the MCP server using a defined standard.

MCP helps accelerate the AI agent building process by giving LLM-powered applications direct access to your analytics data through a defined set of tools. Integrating the BigQuery MCP server with the ADK using the Google OAuth authentication method can be straightforward, as you can see below with our discussion of Agent Development Kit (ADK) and Gemini CLI. Platforms and frameworks such as LangGraph, Claude code, Cursor IDE, or other MCP clients can also be integrated without significant effort.

Let's get started.

Use BigQuery MCP server with ADK

To build a BigQuery Agent prototype with ADK, follow a six-step process:

Prerequisites: Set up the project, necessary settings, and environment.
Configuration: Enable MCP and required APIs.
Load a sample dataset.
Create an OAuth Client.
Create a Gemini API Key.
Create and test agents.

IMPORTANT: When planning for a production deployment or using AI agents with real data, ensure adherence to AI security and safety and stability guidelines.

Step 1: Prerequisites > Configuration and environment

1.1 Set up a Cloud Project
Create or use existing Google Cloud Project with billing enabled.

1.2 User roles
Ensure your user account has the following permissions to the project:

roles/bigquery.user (for running queries)
roles/bigquery.dataViewer (for accessing data)
roles/mcp.toolUser (for accessing MCP tools)
roles/serviceusage.serviceUsageAdmin (for enabling apis)
roles/iam.oauthClientViewer (oAuth)
roles/iam.serviceAccountViewer (oAuth)
roles/oauthconfig.editor (oAuth)

1.3 Set up environment
Use MacOS or Linux Terminal with the gcloud CLI installed.

In the shell, run the following command with your Cloud PROJECT_ID and authenticate to your Google Cloud account; this is required to enable ADK to access BigQuery.

code_block: <ListValue: [StructValue([('code', '# Set your cloud project id in env variable\r\nBIGQUERY_PROJECT=PROJECT_ID\r\n\r\ngcloud config set project ${BIGQUERY_PROJECT}\r\ngcloud auth application-default login'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5c40>)])]>

Follow the prompts to complete the authentication process.

Step 2: Configuration > User roles and APIs

2.1 Enable BigQuery and MCP APIs
Run the following command to enable the BigQuery APIs and the MCP APIs.

code_block: <ListValue: [StructValue([('code', 'gcloud services enable bigquery.googleapis.com --project=${BIGQUERY_PROJECT}\r\ngcloud beta services mcp enable bigquery.googleapis.com --project=${BIGQUERY_PROJECT}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5be0>)])]>

Step 3: Load sample dataset > cymbal_pets dataset

3.1 Create cymbal_pets dataset
For this demo, let’s use the cymbal_pets dataset. Run the following command to load the cymbal_pets database from the public storage bucket:

code_block: <ListValue: [StructValue([('code', '# Create the dataset if it doesn\'t exist (pick a location of your choice)\r\n# You can add --default_table_expiration to auto expire tables.\r\nbq --project_id=${BIGQUERY_PROJECT} mk -f --dataset --location=US cymbal_pets\r\n\r\n# Load the data\r\nfor table in products customers orders order_items; do \r\nbq --project_id=${BIGQUERY_PROJECT} query --nouse_legacy_sql \\\r\n "LOAD DATA OVERWRITE cymbal_pets.${table} FROM FILES(\r\n format = \'avro\',\r\n uris = [ \'gs://sample-data-and-media/cymbal-pets/tables/${table}/*.avro\']);"\r\ndone'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5820>)])]>

Step 4: Create OAuth Client ID

4.1 Create OAuth Client ID
We will be using Google OAuth to connect to the BigQuery MCP server.

7. In the Google Cloud console, go to Google Auth Platform > Clients > Create client

- *Select Application type value as “Desktop app”.
- Once client is created, make sure to copy the Client ID and Secret and keep it safe.

Optional: If you used a different project for OAuth client, run this with your CLIENT_ID_PROJECT

code_block: <ListValue: [StructValue([('code', 'gcloud beta services mcp enable bigquery.googleapis.com --project=CLIENT_ID_PROJECT'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5a90>)])]>

Note [for Cloud Shell Users only]: If you are using Google Cloud Shell or any hosting environment other than localhost, you must create a "Web application" OAuth Client ID.

For a Cloud Shell environment:

For “Authorized JavaScript origins” value use output of this command:
echo "https://8000-$WEB_HOST"
For “Authorized redirect URIs” value use output of this command:
echo "https://8000-$WEB_HOST/dev-ui/"
(URIs in Cloud Shell are temporary and expire after the current session)

Note: If you decide to use a web server, then you will need to use the “Web Application” type OAuth Client and fill in the appropriate domain and redirect URIs.

Step 5: API Key for Gemini

5.1 Create API Key for Gemini
Create a Gemini API key at API Keys page. We will need a generated key to access the Gemini model using ADK.

Step 6: Create ADK web application

6.1 Install ADK
To install ADK and initiate an agent project, follow the instructions outlined in the Python Quickstart for ADK.

6.2 Create a new ADK Agent
Now, create a new agent for our BigQuery remote MCP server integration.

code_block: <ListValue: [StructValue([('code', 'adk create cymbal_pets_analyst\r\n\r\n#When prompted, choose the following:\r\n#2. Other models (fill later)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5850>)])]>

6.3 Configure the env file
Run following common to update the cymbal_pets_analyst/.env file, with the below list of variables and their actual values.

code_block: <ListValue: [StructValue([('code', 'cat >> cymbal_pets_analyst/.env <<EOF\r\nGOOGLE_GENAI_USE_VERTEXAI=FALSE\r\nGOOGLE_CLOUD_PROJECT=BIGQUERY_PROJECT\r\nGOOGLE_CLOUD_LOCATION=REGION\r\nGOOGLE_API_KEY=AI_STUDIO_API_KEY\r\nOAUTH_CLIENT_ID=YOUR_CLIENT_ID\r\nOAUTH_CLIENT_SECRET=YOUR_CLIENT_SECRET\r\nEOF'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5130>)])]>

6.4 Update the agent code
Edit the cymbal_pets_analyst/agent.py file, replace file content with the following code.

code_block: <ListValue: [StructValue([('code', 'import os\r\nfrom google.adk.agents.llm_agent import Agent\r\nfrom google.adk.tools.mcp_tool import McpToolset\r\nfrom google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams\r\nfrom google.adk.auth.auth_credential import AuthCredential, AuthCredentialTypes\r\nfrom google.adk.auth import OAuth2Auth\r\nfrom fastapi.openapi.models import OAuth2\r\nfrom fastapi.openapi.models import OAuthFlowAuthorizationCode\r\nfrom fastapi.openapi.models import OAuthFlows\r\nfrom google.adk.auth import AuthCredential\r\nfrom google.adk.auth import AuthCredentialTypes\r\nfrom google.adk.auth import OAuth2Auth\r\n\r\ndef get_oauth2_mcp_tool():\r\n auth_scheme = OAuth2(\r\n flows=OAuthFlows(\r\n authorizationCode=OAuthFlowAuthorizationCode(\r\n authorizationUrl="https://accounts.google.com/o/oauth2/auth",\r\n tokenUrl="https://oauth2.googleapis.com/token",\r\n scopes={\r\n "https://www.googleapis.com/auth/bigquery": "bigquery"\r\n },\r\n )\r\n )\r\n )\r\n auth_credential = AuthCredential(\r\n auth_type=AuthCredentialTypes.OAUTH2,\r\n oauth2=OAuth2Auth(\r\n client_id=os.environ.get(\'OAUTH_CLIENT_ID\', \'\'),\r\n client_secret=os.environ.get(\'OAUTH_CLIENT_SECRET\', \'\')\r\n ),\r\n )\r\n\r\n bigquery_mcp_tool_oauth = McpToolset(\r\n connection_params=StreamableHTTPConnectionParams(\r\n url=\'https://bigquery.googleapis.com/mcp\'),\r\n auth_credential=auth_credential,\r\n auth_scheme=auth_scheme,\r\n )\r\n return bigquery_mcp_tool_oauth\r\n\r\n\r\nroot_agent = Agent(\r\n model=\'gemini-3-pro-preview\',\r\n name=\'root_agent\',\r\n description=\'Analyst to answer all questions related to cymbal pets store.\',\r\n instruction=\'Answer user questions, use the bigquery_mcp tool to query the cymbal pets database and run queries.\',\r\n tools=[get_oauth2_mcp_tool()],\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57acc5640>)])]>

6.5 Run the ADK application
Run this command from the parent directory that contains cymbal_pets_analyst folder.

code_block: <ListValue: [StructValue([('code', 'adk web --port 8000 .'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c9a6f10>)])]>

Launch the browser, point to http://127.0.0.1:8000/ or the host where you are running ADK, and select your agent name from the dropdown. You now have your personal agent to answer questions about the cymbal pets data. When the agent connects to the MCP server, it will initiate the OAuth flow and you will be able to grant permissions to access.

As you can notice in the second prompt, you no longer need to specify the project id. This is because the agent can infer this information from the conversation.

Here are some questions you can ask:

What datasets are in my_project?
What tables are in the cymbal_pets dataset?
Get the schema of the table customers in cymbal_pets dataset
Find the top 3 orders by volume in the last 3 months for the cymbal pet store in the US west region. Identify the customer who placed the order and also their email id.
Can you get top 10 orders instead of the top one?
Which product sold the most in the last 6 months?

Use BigQuery MCP server with Gemini CLI

To use Gemini CLI, you can use the following configuration in your ~/.gemini/settings.json file. If you have an existing configuration, you will need to merge this under mcpServers field.

code_block: <ListValue: [StructValue([('code', '{\r\n "mcpServers": {\r\n "bigquery": {\r\n "httpUrl": "https://bigquery.googleapis.com/mcp",\r\n "authProviderType": "google_credentials",\r\n "oauth": {\r\n "scopes": [\r\n "https://www.googleapis.com/auth/bigquery"\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c9a6220>)])]>

Then run authenticate with gcloud.

code_block: <ListValue: [StructValue([('code', 'gcloud auth application-default login --clien-id-file YOUR_CLIENT_ID_FILE'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c9a6520>)])]>

Run Gemini CLI.

code_block: <ListValue: [StructValue([('code', 'gemini'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe57c9a6100>)])]>

BigQuery MCP server for your agents

You can integrate BigQuery tools into your development workflow and create intelligent data agents using LLMs and the BigQuery MCP server. Integration is based on a single, standard protocol compatible with all leading Agent development IDEs and frameworks. Of course, before you build agents for production or use them with real data, be sure to follow AI security and safety guidelines.

We are excited to see how you leverage BigQuery MCP server to develop data analytics generative AI applications.

What’s new with Google Data Cloud - 2025

Wed, 31 Dec 2025 17:00:00 +0000

December 15 - December 19

We introduced new self-service capabilities in Looker platform, enabling users to analyze local data alongside governed models, organize complex dashboards more effectively, and align the look and feel of their analytics with corporate branding.
We outlined how you can connect your enterprise data to Google’s new Antigravity IDE. Leveraging the MCP Toolbox for Databases, you can securely connect your AI agents to services like AlloyDB for PostgreSQL, BigQuery, Spanner, Cloud SQL, Looker and others within Google’s Data Cloud, all within your development workflow.
If you want trustworthy AI, what you need is a semantic layer that acts as the single source of truth for business metrics. We demonstrated how you can connect Looker to Gemini Enterprise in minutes, using the MCP Toolbox and Agent Development Kit (ADK).

December 8 - December 12

We introduced Model Context Protocol (MCP) support for Google Services, starting with BigQuery, Maps, Google Compute Engine (GCE) and Google Kubernetes Engine (GKE). This announcement enables AI agents to point to a globally consistent and enterprise-ready endpoint for Google and Google Cloud services. In the next few months, we will roll out managed MCP support for more products, including AlloyDB, Cloud SQL, Spanner, Looker, Pub/Sub and Dataplex Universal Catalog.
We recently launched the preview of data products in Dataplex Universal Catalog, Google Cloud’s unified, intelligent data to AI governance solution. A data product is a curated, ready-to-use package of data assets, documentation, and governance controls, all purposefully assembled to solve a specific business problem. Customers like Virgin Media O2 are using data products in Dataplex to deploy trusted data and scalable AI products.
We recently commissioned a Forrester Consulting Total Economic Impact™ (TEI) study. This report analyzes how building a data lakehouse with Google Cloud’s BigQuery and BigLake enabled organizations to get the flexibility of a data lake with open table formats like Apache Iceberg and the performance and governance of a high performance data warehouse, delivering the best of both on a single, open platform. Get the report to learn more about the ROI of building your data lakehouse with Google Cloud and how it can help you get your data AI-ready.

December 1 - December 5

Last week, we launched AlloyDB AI’s forecasting capabilities. Traditionally, performing high-quality time-series forecasting (for sales, demand, inventory, etc.) required moving data to external platforms and engaging in long, complex model training and validation cycles. The new AI.FORECAST function solves this by bringing state-of-the-art predictive power directly into your operational database with a single SQL command.
We announced the General Availability of Conversational Analytics in Looker, enabling all organizations to ask questions of of their data in natural language and get insightful answers, powered by Gemini. We followed up with the addition of Looker and Looker Conversational Analytics extensions in Gemini’s CLI.
We held our Looker Innovation roadmap on December 4th. Hosted by Google Cloud product management and engineering leaders, you can view the recording on demand to learn the latest updates on Conversational Analytics and see how Looker is expanding self-service capabilities.
We are making it easier for agent developers in Google’s Agent Development Kit (ADK) to answer questions. We are introducing BigQuery Agent Analytics, a new plugin for ADK that exports your agent's interaction data directly into BigQuery to capture, analyze, and visualize agent performance, user interaction, and cost.
We are thrilled to announce the General Availability of AlloyDB AI’s columnar engine powered vector search. For critical applications like medical imaging, fraud detection, and legal research, 100% accuracy (perfect recall) is non-negotiable, meaning Approximate Nearest Neighbor (ANN) search is insufficient. However, running exact K-Nearest Neighbor (KNN) searches across large datasets often results in high latency, blocking real-time, mission-critical similarity searches.

November 17 - November 21

This week, we launched a new, dedicated Cloud SQL Free Trial Instance program offering 30 days of risk-free access to Enterprise Plus features, including High Availability and Data Cache, on a powerful 8vCPU/64GB N2 instance. This is your chance to validate mission-critical performance for MySQL and PostgreSQL with zero operational commitment. (Docs)
Our customers are leveraging the full power of Google Cloud's database portfolio to drive major business and technical breakthroughs. Companies like Pager Health and Dun & Bradstreet are using Cloud SQL and GKE to unify their infrastructure, reducing complexity while delivering world-class healthcare and risk solutions. For massive scale and emerging challenges, ID.me chose AlloyDB to handle 10TB+ workloads and power their generative AI projects with validated data to fight fraud for 145 million users. Finally, Palo Alto Networks built their globally distributed, high-availability security platform on Spanner and Spanner Graph, creating a robust data foundation for their critical, AI-driven workflows.

November 3 - November 7

We have announced the next generation of Spanner-better-with-BigQuery capabilities delivering streaming insights, faster federated queries, cross-region data operations across Spanner and BigQuery data including Iceberg tables.
Memory Agent for Cloud SQL for PostgreSQL is now generally available. Previously, memory-intensive queries could cause PostgreSQL restarts due to the Linux OOM killer. This led to downtime and no clear way for users to identify problematic queries. The new Memory Agent proactively detects and gracefully cancels high-memory connections, preventing restarts. With a recommender, it offers details and suggestions to alleviate memory pressure, providing a better user experience.
We're excited to announce the General Availability of Customer-managed Active Directory integration with Cloud SQL for SQL Server. This allows Windows authentication for Cloud SQL for SQL Server instances using existing AD environments, eliminating the need for Google Managed AD and simplifying critical SQL Server workloads.

October 24 - October 31

Dive into the newest Google Cloud Tech Bytes videos for Cloud SQL and Spanner! Get the practical details you need to set up and optimize our fully managed databases so you can simplify operations and accelerate development.

October 20 - October 24

Database Migration Service now offers Object Level Observability, providing enhanced visibility and control over data migration. Previously limited to job-level oversight, these capabilities have been expanded to the individual table level, allowing for detailed insight into your data movement while heterogeneous database migration (e.g SQL Server to PostgreSQL).
Cloud SQL's Enterprise Plus edition now supports the Axion-based C4A machine series in GA. This offers our customers significant performance benefits: nearly 50% better price-performance compared to current N2 machines and up to 2x greater transactional throughput than Amazon RDS Graviton 4-based offerings.
Firestore with Enterprise Edition now offers Saved Queries.This new feature enables users to save and share queries for a specific database directly from the Firestore Studio page.
At Oracle AI World ‘25, Database Center announced expanded support for Oracle Database@Google Cloud. This update allows customers to monitor Oracle Exadata and Autonomous databases, including their inventory and metrics, directly within the Database Center UI and Chat. Now, Google Cloud database services and Oracle inventory can be monitored side-by-side.
Managed Kafka Connect is now generally available. Replicate on-prem clusters to Managed Service for Apache Kafka clusters, surface Kafka data in BigQuery, backup the data in Cloud Storage, or activate it in Pub/Sub. Unlock the real value of your Kafka data. Get started with Kafka Connect today.
October 13 - October 17
Don't miss the Database Innovation Roadmap Webinar on October 30th, where we'll reveal the strategies and roadmap to supercharge agentic development and the next wave of AI innovation. This event kicks off our new Database Innovation Series, granting you access to 5+ deep-dive sessions shortly after the main event!

October 6 - October 10

Cloud SQL now offers point-in-time recovery (PITR) for deleted instances, addressing compliance, accidental deletion, and disaster recovery needs. This feature requires customers to enable backup retention and PITR on their instances. Users can utilize the existing PITR clone API (with source-instance-deletion-time) and getLatestRecoveryTime API to manage deleted instances. The PITR window shortens based on log retention: up to 35 days for Enterprise Plus instances and 7 days for Enterprise instances.
Introducing the Precheck API for Cloud SQL for PostgreSQL. This new feature improves Major Version Upgrades by proactively identifying potential issues, preventing unplanned downtime caused by instance incompatibilities (extensions, flags, data types). It addresses customer requests for a precheck utility to identify and remedy upgrade issues beforehand.
AlloyDB now supports the tds_fdw extension, enabling direct access to SQL Server and Sybase databases. This feature streamlines database migrations and allows hybrid data analysis, complementing existing oracle_fdw support.

September 29 - October 3

Cloud SQL announced support for Managed Connection Pool (in GA) across MySQL and PostgreSQL

- Managed Connection Pooling lets you scale your workloads by optimizing resource utilization for Cloud SQL instances using pooling. You can now also use IAM authentication to secure connections when using Managed Connection Pooling. To understand how it works, its key benefits, and how to configure Managed Connection Pooling for your workloads, dive into these guides:
  - MySQL: https://discuss.google.dev/t/boost-your-cloud-sql-for-mysql-performance-through-managed-connection-pooling/269283
  - PostgreSQL: https://discuss.google.dev/t/optimizing-performance-and-scaling-with-managed-connection-pooling-for-cloud-sql-for-postgresql/270528?u=sagarsidhpura

September 22 - September 26

AlloyDB now supports PostgreSQL 17 in GA

AlloyDB now offers general availability for PostgreSQL 17, bringing with it a range of new features and significant enhancements. Key improvements include:

Improved query performance, particularly for materialized Common Table Expressions
Incremental backup capabilities
Enhanced logical replication features
Improvements to the JSON data type handling

Build AI Agents with Enterprise Databases (NEW! Training Course)

This on-demand course teaches how to build AI agents that can leverage our enterprise databases using MCP Toolbox for Databases, as a secure middle layer. You will learn to securely connect AI agents to your existing databases like AlloyDB, Cloud SQL, and Spanner. You can define secure database interaction tools and implement intelligent querying capabilities, including semantic search with vector embeddings.

Gemini CLI extensions for Data Cloud services and popular open source databases released

In June, Google launched the open-source Gemini CLI. Now, developers can leverage open-source Gemini CLI extensions for Google Data Cloud services such as Cloud SQL, AlloyDB, and BigQuery. These extensions streamline data interactions and enhance application development directly from their local environment. For more details, check out the extensions documentation. You can also explore existing templates to begin creating and sharing your own extensions with the community.

Cloud SQL for PostgreSQL now supports the pg_roaringbitmap extension

Cloud SQL developers will now benefit from the ability to handle high-scale analytics, complex filtering, and large set operations directly within the managed PostgreSQL environment with unprecedented speed and efficiency.

September 15 - September 19

Benchmark-Driven Kafka Optimization: Maximize Throughput and Cut Costs on Google Cloud
- Choosing the right compression strategy for Google Cloud Managed Service for Kafka is one of the most critical decisions impacting your performance and budget—and many are leaving massive savings on the table. Relying on default settings or guesswork can lead to unnecessarily high network and storage costs, increased latency, and severe throughput bottlenecks. This new, in-depth guide moves beyond theory to provide hard benchmark data, empowering you to make data-driven decisions.This comprehensive analysis systematically tests the most popular codecs (including GZIP, SNAPPY, and LZ4) against a "no compression" baseline. Read the full guide and get the sample benchmark code here.
Explore and experiment with Spanner's advanced capabilities with ease. Say goodbye to friction and hello to innovation.
- Free 90-day trial
- Pre-loaded datasets for retail, banking, finance, and more
- Easy data import from MySQL, PostgreSQL dump files, and CSV
- Dozens of sample queries showcasing advanced features like full-text search, vector search, and graph capabilities
C4A Axion processor support is now in GA for AlloyDB
- It was launched in Preview during Next'25. Customers waiting for GA to evaluate / onboard for production can now get better performance, price-performance and can run their development environment with 50% reduced entry price using one vCPU. Ready to get started? If you’re new to AlloyDB, You can sign-up via the AlloyDB free trial link.
Parameterized Secured Views (now in Preview) in AlloyDB provides application data security and row access control using SQL views.

September 8 - September 12

From query to cart: Inside Target’s search bar overhaul with AlloyDB AI
- Target set out to modernize its digital search experience to better match guest expectations and support more intuitive discovery across millions of products. To meet that challenge, they rebuilt their platform with hybrid search powered by filtered vector queries and AlloyDB AI. Target achieved faster, smarter, more resilient search experience that’s already improved product discovery relevance by 20% and delivered measurable gains in performance and guest satisfaction.
Powering smarter recommendations with Bigtable and BigQuery
- Schibsted Marketplaces, a leading online classifieds group in the Nordic region, cut infrastructure costs by 70% and accelerated data insights and model development by adopting Bigtable and BigQuery. This led to faster, more relevant recommendations and a better user experience.
AlloyDB AI natural language support launched in Public Preview
- AlloyDB now simplifies the process for enterprises to develop highly accurate and secure Gen AI applications. These applications enable end-users to interact with their own data using natural language. The new natural language APIs integrate seamlessly into agentic architectures and are compatible with Gen AI orchestration frameworks like LangChain, making real-time operational data more accessible for end-user-facing chat experiences.
Cloud SQL announced support for the Read Pools (in GA) across MySQL and PostgreSQL
- Cloud SQL's read pools offer a significant advantage over self-managed databases, particularly for read-heavy workloads. They simplify operations and enhance scalability by providing a single endpoint for up to 20 read pool nodes, automatically balancing traffic among them. Read pools can also be dynamically scaled up, down, out, or in to accommodate traffic surges.

August 25 - August 29

Firestore with MongoDB compatibility is now generally available (GA)
- Developers can now build cost-effective, scalable, and highly reliable apps on Firestore's serverless database using a familiar MongoDB-compatible API. With the general availability of Firestore with MongoDB compatibility, the 600,000 active developers within the Firestore community can now use existing MongoDB application code, drivers, and tools, as well as the open-source MongoDB ecosystem, with Firestore's serverless service. Firestore offers benefits like multi-region replication, virtually unlimited scalability, up to 99.999% SLA, single-digit millisecond read performance, integrated Google Cloud governance, and pay-as-you-go pricing.
- Register now for an exciting webinar on September 9th for a deep dive into Firestore with MongoDB compatibility and see live demos.
Database Migration Service (DMS) offers support for Private Service Connect (PSC) interfaces for homogenous migrations to Cloud SQL (PSCi support doc) and AlloyDB (PSCi support doc). This capability is now generally available (GA).

August 18 - August 22

Simplify Data Ingestion with the Revamped BigQuery "Add Data" Experience
- We're excited to announce the general availability of a completely redesigned "Add Data" experience in BigQuery, built to streamline how you bring data in for analysis.
- To enhance the user journey, we focused on simplifying the process of choosing from the many powerful ingestion methods BigQuery supports, from batch and streaming to CDC. Our goal was to create a more intuitive path for discovering data sources and provide clearer guidance on selecting the right tool for any given task.
- The new "Add Data" experience achieves this with a single, unified starting point within BigQuery Studio. It brings together all the ways to get data into BigQuery—including Data Transfer Service, Datastream, Dataflow, and partner solutions—into one intuitive interface. The experience guides you with clear categorization, solution recommendations, and in-context documentation to help you make informed choices. Now you can easily discover and configure the right data pipeline for your needs without leaving the BigQuery console.
- Get started by clicking the "+ Add data" button in the BigQuery Explorer pane today. Learn more in the official documentation.
Cloud SQL now supports Private Service Connect (PSC) outbound connectivity
- With PSC outbound connectivity, customers can attach a PSC interface to their existing Cloud SQL PSC-enabled instances to allow their instances to make outbound connections to their network. This is required for homogeneous migrations using Database Migration Service. For more information, see PSC outbound connections.
AI-Assisted Troubleshooting in Cloud SQL Enterprise Plus
- Cloud SQL for Enterprise Plus edition now offers enhanced AI-assisted troubleshooting, guiding you through resolving complex database performance issues such as slow queries and high load on your instances. This feature requires Gemini Cloud Assist and query insights, both available with the Enterprise Plus edition.

August 11 - August 15

Code Your Way to $15,000: The BigQuery AI Hackathon Starts Now - go beyond traditional analytics and build groundbreaking solutions using BigQuery's cutting-edge AI capabilities. This is your opportunity to solve real-world business problems using BigQuery’s Generative AI, Vector Search, and Multimodal capabilities. You’ll get hands-on experience with BigQuery’s newest features that bring AI directly to your data. SQL users will find these capabilities feel like a natural extension of their existing workflow, while Python practitioners can use BigQuery DataFrames to work using a familiar, pandas-like API. The goal is simple: build powerful, scalable AI solutions right where your data lives. Sign-up today!
AlloyDB now supports PG 17 (17.5 minor version) in Preview - AlloyDB customers can now access the latest improved version of Postgres, alongside existing versions like PG16, PG15, and PG14. Customers will also be able to upgrade to PG17 through MVU APIs. The community released PG17 in September 2024, introducing numerous new features and improvements. These include enhanced query performance (materialized Common Table Expressions, incremental backups and improved logical replication), a better developer experience (enhancements to the JSON support) and numerous other improvements.
Database Center now supports self-managed databases on GCE - Back in April, we announced the general availability of Database Center, your AI-powered unified fleet management solution for Google Cloud databases including Cloud SQL, AlloyDB, Spanner, Bigtable, Memorystore, and Firestore. However, many of our customers continue to leverage the flexibility of running their Postgres, MySQL and SQL server databases on Google Compute Engine (GCE) VMs. So we're thrilled to announce that Database Center now extends its monitoring capabilities to these self-managed databases. Please sign-up here to join this preview phase.
Near Zero Downtime (nZDT) for Cloud SQL Enterprise Plus edition for SQL Server is now GA - With nZDT, maintenance and machine tier upgrades for Enterprise Plus SQL Server instances now experience sub-second downtime. This means:
- 99.99% SLA now includes maintenance downtime.
- Customers can say goodbye to lengthy planning cycles for maintenance.
- nZDT is now available across all three Cloud SQL engines - SQL Server, PostgreSQL and MySQL.
Database Clone Feature in Firestore launched in Public Preview - Firestore database cloning allows Firestore users to create a copy of their database. All the Firestore Documents data, as well as index definitions and entries, are copied over to a new database in the same project & region with an appropriate user-chosen new database name. The user may choose to copy the state of the database from any snapshot time up to 7 days in the past.
Build with Google Databases: 70+ Success Stories - This powerful resource highlights how over 70+ companies are using Google Cloud's fully managed database services to improve performance, scale globally, and optimize costs. It showcases real-world success stories across 10 industries, including retail, financial services, and technology.

August 4 - August 8

Next Tokyo Data Cloud Announcements - Google’s Data Cloud gives agents a complete, real-time understanding of your business, transforming it into a self-aware, reliable organization. We're delivering key innovations in three areas: 1) A new suite of data agents to act as expert partners, 2) An interconnected network for seamless agent collaboration, 3) A unified, AI-native foundation that unifies data and embeds AI-driven reasoning.
AI-first Colab Enterprise experience in Vertex AI and BigQuery: This powerful platform streamlines complex data science workflows, allowing you to simply prompt an agent with a request like "train a model to predict income." The agent then autonomously generates and executes a complete plan—from data loading and cleaning to model training and evaluation
Spanner Columnar Engine: Announcing the preview of the Spanner columnar engine, our latest innovation designed to turbocharge your data. By combining columnar storage and vectorized execution, we're making it possible to run lightning-fast analytical queries directly on your live operational data.
Enhanced Backups for Cloud SQL: Introducing Enhanced Backups for Google Cloud SQL, now with logically air-gapped and immutable backup vaults. Built with Google Cloud Backup and DR Service, this is your ultimate defense against modern threats.

July 28 - August 1

AlloyDB Omni now supports Kubernetes Operator 1.5.0 and PostgreSQL ver. 16.8.0/15.12.0: We have launched AlloyDB Omni Operator 1.5.0 and database versions 16.8.0/15.12.0. This major release delivers a significant step forward in enterprise readiness, including support for OpenShift operations, high availability/disaster recovery, and critical operational improvements like low-downtime upgrades and backups from standby.

July 21 - July 25

Introducing partitioned index for BigQuery vector search: When creating a vector index on a partitioned BigQuery table, you now have the option to also partition your vector index. Partitioning your vector index significantly reduces query costs and improves search accuracy for vector searches that utilize pre-filtering on the partitioning column.By partitioning your vector index, BigQuery can apply partition pruning to both your table and your vector index when you use a filter on the partitioning column in your vector search. This means BigQuery only scans the relevant partitions, decreasing I/O costs. Additionally, pre-filtering on the partitioning column makes your vector searches less likely to miss relevant results. This feature is particularly beneficial if most of your vector searches target specific partitions using pre-filters. You can only partition TreeAH vector indexes, and the PARTITION BY clause used for the vector index must match the one used for the original table. Read more about the partitioned indexes in vector search.
Datastream now supports BigLake Iceberg tables in BigQuery: Customers can now easily replicate data from different supported sources (MySQL, Postgres, SQLserver, Oracle,Salesforce and MongoDB ) of Datastream into BigLake Managed Tables for use cases spanning across open lakehouse, Enterprise grade storage for analytics, streaming and AI. Streaming to BigLake Iceberg tables lets you store data in a cost-effective way in the PARQUET format. By doing this, you can keep your data in a Cloud Storage bucket while using BigQuery for querying and analysis.
Cloud SQL Write Endpoint for Advanced DR: Cloud SQL is excited to announce the GA of Write Endpoint to make Advanced Disaster Recovery (DR) seamless for customers (Documentation). This feature enhances application resilience during instance failovers and switchovers, ensuring customer applications remain connected to the primary instance without manual intervention.The write endpoint is now available in GA for MySQL and PostgreSQL instances of Enterprise Plus Edition. It already exists for SQL Server instances.
Vertical Scaling for Memorystore for Valkey and Memorystore for Redis Cluster: Using Vertical Scaling, Memorystore customers can now effortlessly scale their Memorystore nodes up or down ensuring optimal cluster sizing for varying workloads. Previously, node types were immutable post-deployment, hence customers only had the option for horizontal scaling (in and out) changing the number of shards in the cluster.
Database Migration Service (DMS) supports migrations from SQL Server to AlloyDB for PostgreSQL in GA: Customers can now use DMS to migrate their databases from SQL Server to AlloyDB for PostgreSQL . This migration offers seamless experience, which offers a comprehensive SQL Server modernization framework with:
- Automatic database schema and code conversion
- Gemini augmented database code conversion
- Gemini assisted PostgreSQL training and code improvements
- Low-downtime, CDC based data movement

July 14 - July 18

Trust and security are central to Conversational Analytics. Designed to gain the benefits of Google’s most capable AI models, Conversational Analytics offers a powerful and insightful natural language experience that is secure and trustworthy, meaning you can realize the full potential of generative AI with confidence, while keeping your data under control. Learn more here.
Turn questions into queries with the Conversational Analytics API. The Conversational Analytics API, now in preview, integrates multiple AI-powered tools to process user requests, including Natural Language to Query (NL2Query) and a Python code interpreter for generating responses, simplifying data science. Learn more here.
Introducing BigQuery Soft Failover: Greater Control Over Disaster Recovery. BigQuery now offers "soft failover," giving administrators options over failover procedures. Unlike "hard failover" for unplanned outages, soft failover minimizes data loss for planned activities like disaster recovery drills or workload migrations. It initiates failover only after all data is replicated to the secondary region, guaranteeing data integrity. This feature is available via BigQuery UI, DDL, and CLI, providing enterprise-grade control for disaster recovery, confident simulations, and compliance without risking data. Learn more here.

July 7 - July 11

[Webinar] Join us for a session on "Build Smart Apps with Ease: Gen AI, Cloud SQL, and Observability for Faster Development." This webinar dives deep into mastering the essentials of building powerful Gen AI applications using Google Cloud technologies. Discover the complete Gen AI application development lifecycle, get a live demonstration of the new Application Design Center (ADC) for rapid app deployment, and explore its seamless integrations with frameworks like LangChain, LlamaIndex, and LangGraph. Plus, learn about the new MCP Toolbox for Databases to enhance the manageability and security of your GenAI agents, and understand critical operational considerations, including Cloud SQL Enterprise Plus features for performance, scalability, high availability, and disaster recovery.

June 23 - June 27

Looker developers gain speed and accuracy with debut of Continuous Integration. Continuous Integration for Looker helps streamline code development workflows, boost the end-user experience, and gives developers the confidence to deploy changes faster. Learn more here.
Code Interpreter brings advanced data science capabilities to Conversational Analytics. Code Interpreter helps answer complicated questions, tapping into Python to perform advanced analysis on your Looker data. Learn more here.

June 16 - June 20

Standardize your business terminology with Dataplex business glossary. Want to standardize business terminologies and build a shared understanding across the enterprise? Dataplex business glossary is now GA within Dataplex Universal Catalog, providing a central, trusted vocabulary for your data assets, streamlining data discovery, and reducing ambiguity — leading to more accurate analysis, better governance, and faster insights. Learn more here.
Looker Core on Google Cloud is now FedRAMP High authorized. The need to protect highly sensitive government data is a top priority. Looker Core on Google Cloud enables users to explore and chat with their data via AI agents using natural language, and create dashboards and self-service reports. Learn more here.
Fast Dev Mode Transition Speeds Looker Developers. A new Labs feature, Fast Dev Mode Transition, improves the performance of Development Mode on your Looker instance by loading LookML projects in read-only mode until a developer clicks the Create Developer Copy button for the project. Learn more here.
Datastream now supports MongoDB as a Source (in Public Preview): You can now easily replicate data from MongoDB source into BigQuery and Cloud Storage for advanced analytics, reporting, and to power generative AI applications. Datastream offers MongoDB connectivity for both Replica Sets and Sharded Clusters. This includes support for self-managed MongoDB deployments as well as the fully managed AtlasDB service.
Private Service Connect (PSC) on existing Cloud SQL instances (GA): Cloud SQL now offers the ability to enable Private Service Connect (PSC) on existing instances that currently utilize Private Service Access (PSA). This new functionality, generally available for PostgreSQL, MySQL, and SQL Server engines, eliminates the previous requirement of creating new instances for PSC adoption. Customers can now transition their existing PSA instances to PSC without data migration.
Cloud SQL for SQL Server - E+ Recommender: The Enterprise Plus recommender helps customers identify SQL Server instances that would benefit from an upgrade to the Cloud SQL Enterprise Plus Edition. It offers insights into current performance metrics, and emphasizes how Enterprise Plus features (such as the data cache and memory-optimized machines) can boost performance. Additionally, the recommender includes a convenient button for direct navigation to the instance settings page, enabling users to perform the upgrade easily.
AlloyDB - PSC Service Automation: With this launch, AlloyDB significantly improves the connectivity configuration experience for Private Service Connect (PSC), by automatically creating PSC endpoints in the customer VPC and exposing the IP address of the endpoint directly through the AlloyDB API, enabling seamless PSC adoption at scale.

June 9 - June 13

Introducing Pub/Sub Single Message Transforms (SMTs), to make it easy to perform simple data transformations such as validate, filter, enrich, and alter individual messages as they move in real time right within Pub/Sub. The first SMT is available now: JavaScript User-Defined Functions (UDFs), which allows you to perform simple, lightweight modifications to message attributes and/or the data directly within Pub/Sub via snippets of JavaScript code. Learn more in the launch blog.
Serverless Spark is now generally available directly within BigQuery. Formerly Dataproc Serverless, the fully managed Google Cloud Serverless for Apache Spark helps to reduce TCO, provides strong performance with the new Lightning Engine, integrates and leverages AI, and is enterprise-ready. And by bringing Apache Spark directly into BigQuery, you can now develop, run and deploy Spark code interactively in BigQuery Studio. Read all about it here.
Next-Gen data pipelines: Airflow 3 arrives on Google Cloud Composer: Google is the first hyperscaler to provide selected customers with access to Apache Airflow 3, integrated into our fully managed Cloud Composer 3 service. This is a significant step forward, allowing data teams to explore the next generation of workflow orchestration within a robust Google Cloud environment. Airflow 3 introduces powerful capabilities, including DAG versioning for enhanced auditability, scheduler-managed backfills for simpler historical data reprocessing, a modern React-based UI for more efficient operations, and many more features.

June 2 - June 6

Enhancing BigQuery workload management: BigQuery workload management provides comprehensive control mechanisms to optimize workloads and resource allocation, preventing performance issues and resource contention, especially in high-volume environments. To make it even more useful, we announced several updates to BigQuery workload management around reservation fairness, predictability, flexibility and “securability,” new reservation labels, as well as autoscaler improvements. Get all the details here.
Bigtable Spark connector is now GA: The latest version of the Bigtable Spark connector opens up a world of possibilities for Bigtable and Apache Spark applications, not least of which is additional support for Bigtable and Apache Iceberg, the open table format for large analytical datasets. Learn how to use the Bigtable Spark connector to interact with data stored in Bigtable from Apache Spark, and delve into powerful use cases that leverage Apache Iceberg in this post.
BigQuery gets transactional: Over the years, we’ve added several capabilities to BigQuery to bring near-real-time, transactional-style operations directly into your data warehouse, so you can handle common data management tasks more efficiently from within the BigQuery ecosystem. In this blog post, you can learn about three of them: efficient fine-grained DML mutations; change history support for updates and deletes; and real-time updates with DML over streaming data.
Google Cloud databases integrate with MCP: We announced capabilities in MCP Toolbox for Databases (Toolbox) to make it easier to connect databases to AI assistants in your IDE. MCP Toolbox supports BigQuery, AlloyDB (including AlloyDB Omni), Cloud SQL for MySQL, Cloud SQL for PostgreSQL, Cloud SQL for SQL Server, Spanner, self-managed open-source databases including PostgreSQL, MySQL and SQLLite, as well as databases from other growing list of vendors including Neo4j, Dgraph, and more. Get all the details here.