Rakuten Taiwan's BEATS Tags 5.4M Products and Boosts Search Accuracy by 5.7%
Rakuten Taiwan used LLMs and human annotators to build a product attribute taxonomy from scratch, boosting search retrieval by over 5% across 5.4 million products
- BEATS is a deployed human-in-the-loop LLM framework that builds product attribute taxonomies from scratch for e-commerce platforms that lack structured catalog data.
- The pipeline combines multi-stage LLM generation with developer quality checks and domain-expert human annotation, iterated across rounds to refine prompts and improve output quality.
- Attribute acceptance rate jumped from 89% in Round 1 to 99.88% in Round 2 purely through iterative prompt refinement based on annotator feedback.
- Dense retrieval gains: Recall@10 +5.7%, NDCG@10 +5.7%, MRR@10 +5.4% -- all without changing the retrieval model architecture.
- Deployed at Rakuten Taiwan, enriching 9 major categories, 2,694 sub-categories, 67,277 attributes, and over 5.4 million tagged products.
- No code or model weights are publicly released; this is a production system paper describing an operational pipeline.
Most e-commerce search teams spend years curating the structured product data that powers filters like "material: cotton" or "screen size: 15 inches." For platforms in emerging markets, that infrastructure often simply does not exist. E-commerce platforms in these markets frequently operate with product catalogs that contain only broad category hierarchies but lack structured attribute schemas entirely -- and that absence directly limits search capabilities, preventing faceted filtering, degrading query understanding, and weakening the semantic representations search systems rely on.
The cold-start problem for product catalogs
To understand why this is hard, consider what "structured attributes" actually means. Well-structured catalogs with rich attribute taxonomies -- think brand, material, size, and color per product category -- enable faceted navigation, precise filtering, and high-quality semantic representations. Faceted search is a system of filters that allows users to refine product listings based on these attributes; instead of scrolling endlessly, shoppers can define what they are looking for and move directly to relevant results.
Building attribute taxonomies manually is prohibitively expensive across thousands of categories. You need domain experts who understand that a "running shoe" needs attributes like "drop height" and "pronation support," while a "laptop" needs "RAM" and "refresh rate." Multiply that across tens of thousands of sub-categories in a language like Traditional Chinese, and the manual path becomes a non-starter. LLMs offer a promising path for automated taxonomy construction, but deploying LLM-generated structured data raises quality concerns: individual LLMs exhibit biases and can produce outputs that are inconsistent or culturally misaligned.
What BEATS does
The Rakuten Taiwan team presents BEATS, a human-in-the-loop LLM framework for bootstrapping product attribute taxonomies entirely from scratch. The key insight is that LLM generation alone is not enough -- you need a structured feedback loop between automated generation, developer quality checks, and domain-expert human annotators to progressively improve output quality across multiple rounds.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
