Apple scientists carried out an examination to assess the effects of AI-generated relevance labels on App Store search rankings and application downloads. The study, named “Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments,” sought to establish whether large language models (LLMs) could improve search outcomes by generating relevance labels that inform the ranking framework.
The research underscored two main elements affecting search rankings: behavioral relevance, which evaluates user interactions with search results (such as application taps and downloads), and textual relevance, which looks at how well an app’s metadata corresponds with a user’s search query. While behavioral relevance data is plentiful and straightforward to measure, textual relevance labels produced by human reviewers are limited and expensive to create, resulting in a scalability bottleneck.
To tackle this challenge, the researchers optimized a 3-billion-parameter LLM on existing human evaluations, allowing it to create relevance labels based on user queries and app metadata. They generated millions of new relevance labels and recalibrated the App Store ranking system using both the original and LLM-generated datasets.
After the recalibration, the researchers performed an offline assessment and an international A/B test using live App Store traffic. The findings revealed that the LLM-enhanced model recorded a statistically significant 0.24% rise in conversion rate, defined as the fraction of search sessions leading to at least one app download. This enhancement was detected in 89% of storefronts.
Even though a 0.24% increase might appear negligible, it has the potential to yield substantial boosts in app downloads, with forecasts predicting 38 billion total App Store downloads by 2025. This could result in millions of extra downloads, providing significant benefits to app developers.
For more information, the complete study can be accessed [here](https://machinelearning.apple.com/research/augmenting-app).