loading
loading
Announcing Native BM25 Ranking in AlloyDB and Cloud SQL Darshana Sivakumar Group Product Manager Nisa Meshal Customer Engineer Vector search is a critical component of generative AI, retrieval-augmented generation (RAG), and data agent architectures, but sometimes vector search alone isn't enough. While vector embeddings are incredible at understanding conceptual meaning, they stumble on specific alphanumeric IDs and exact product SKU numbers. To build truly robust search and AI applications, you may need the combination of semantic vector search and traditional exact keyword full-text search — what we call hybrid search. In search, Best Matching 25, or BM25, is a key algorithm used to estimate how relevant a document is to a given query. Until today, if you wanted BM25 ranking with AlloyDB or Cloud SQL, you needed to add an additional full-text search backend. This introduced data silos, sync lags, and operational complexity. Today, we are eliminating the friction of maintaining a separate full-text search backend altogether, with the preview of the native BM25 index in AlloyDB and Cloud SQL for PostgreSQL 17+, made possible through the open-source pg_textsearch extension created by Tiger Data. Now, with a unified hybrid search backend, you no longer need to provision, manage, or pay for separate systems to get state-of-the-art full-text retrieval. It all happens directly inside your database, where your operational data lives, delivering: - Industry-standard keyword ranking: Powered by Tiger Data's pg_textsearch , bring lightning-fast, C-optimized BM25 scoring directly to your Postgres tables. - No complexity, total consistency: Eliminate the data duplication, ETL pipelines, and synchronization lag that you get when you maintain multiple backends for vector and full-text retrieval. - Supercharged semantic search (AlloyDB exclusive): Get up to 6x and 10x faster vector search queries (when compared to standard PostgreSQL) with ScaNN and HNSW index types. Why pg_textsearch? If you’ve used PostgreSQL's built-in ts_rank for full-text search at any meaningful scale, you already know its limitations. Ranking quality degrades as your corpus grows. There’s no support for inverse document frequency, so common words carry the same weight as rare ones. There’s no term-frequency saturation, so a document that mentions "database" 50 times outranks one that mentions it once. BM25 is the information retrieval gold standard, providing inverse document frequency (rarer terms matter more), term frequency saturation (repetition doesn't dominate), and document length normalization. You can learn more in this blog post by Tiger Data about how they built a BM25 search engine on PostgreSQL pages. Full-text search example Here’s how to get started with BM25 full-text search on both AlloyDB and Cloud SQL. Consider a sample table, cymbal_products, that contains the unique identifier uniq_id, a product_name column, a product_description column containing a text description of each product, and a generated product_embedding column. cymbal_products contains information on various retail products, including indoor and outdoor plants. Index creation To use BM25, enable the pg_textsearch extension. Create the index on the product_description column from the cymbal_products table. A BM25 full-text search query can be executed using the <@> special operator. In the snippet below, we search for ‘cherry tree’. Sample output is shown below. A more negative score indicates a stronger relevance match. AlloyDB hybrid search example Setting up a hybrid search system in AlloyDB is simple. You can create both your vector and keyword indexes on the same table and merge the results seamlessly using the hybrid search user-defined function (UDF). Vector index creation Here is how to create a ScaNN vector search index: Hybrid search AlloyDB provides an out-of-the-box hybrid search UDF that makes it very simple to run hybrid search queries. The UDF merges the ranked results from each search component into a single, unified list using the Reciprocal Rank Fusion (RRF) algorithm. This query utilizes the UDF to perform a vector search for ‘trees that grow taller than houses’ and a keyword search for ‘California’ in the product description. As shown in the sample output below, results are ranked in descending order of their RRF scores. Here, hybrid search bridges the gap between semantic intuition and exact keyword matching. While vector embeddings excel at grasping conceptual queries, like "trees that grow taller than houses", traditional full-text search provides the pinpoint precision needed for strict identifiers like "California." By fusing the two, AlloyDB helps ensure your application prioritizes highly specific, locally relevant results like ‘California Sycamore’ right at the top of the list. Cloud SQL hybrid search example In Cloud SQL, you can create both your vector and keyword indexes on the same table and merge the results seamlessly using Common Table Expressions (CTEs) and coalescing the RRF score, as shown below. Vector index creation Here is how to create an HNSW index in Cloud SQL. Hybrid search Here is the hybrid search query. The resulting output is identical to the AlloyDB hybrid search results shown above. Watch it in action Watch how this all comes together in this demo video. Relevant resources We are incredibly excited to work with Tiger Data and cannot wait to see how you leverage native BM25 support to build faster, smarter, and simpler AI applications. Turn on the pg_textsearch extension today, and experience the ultimate hybrid search engine experience with AlloyDB and Cloud SQL. Want to get started? Check out” - AlloyDB resources - New to AlloyDB? Discover AlloyDB with a 30-day free trial - Choose a vector index in AlloyDB AI - AlloyDB BM25 documentation - AlloyDB hybrid search UDF documentation - Cloud SQL resources - Tiger Data pg_textsearch Release Page
document fetched 2026-09-19 02:56 · canonical https://cloud.google.com/blog/products/databases/native-bm25-search-in-alloydb-and-cloud-sql