Challenge

Programa, an online project management tool for interior designers and architects, faced a rapidly growing issue with duplication in its product catalogue. The platform enabled users to add products from various brands and sources, including manual entries by designers, web imports, and direct supplier uploads. This influx led to thousands of new product entries daily, many of which were duplicates. The presence of these duplicates significantly impacted user experience, search accuracy, and analytics capabilities. To address this, Programa required a scalable and accurate deduplication solution. DiUS was enlisted to help create a Master Product Catalogue by deduplicating existing entries using machine learning. However, ensuring the ongoing accuracy of the deduplication process demanded a solution that minimised manual intervention and could scale alongside the platform’s growth.

Solution

DiUS implemented a sophisticated approach combining traditional machine learning with Large Language Models (LLMs). A multi-modal model using both text and image data was developed and processed through Amazon SageMaker, creating an efficient data pipeline. This pipeline indexed the cleaned product list in Amazon OpenSearch. For ongoing evaluation, DiUS integrated Claude, an LLM hosted on Amazon Bedrock, to automatically assess the deduplication results. Amazon Bedrock was chosen for its ease of use, allowing for rapid experimentation across multiple LLMs and replacement of models as newer more performant and cost-effective models such as Claude 3.5 emerge.

Results

The automated evaluation process using an LLM gave Programa the confidence to deploy the ML-powered deduplication solution to production. After achieving accurate results with Claude, the manual verification efforts were drastically reduced from several hours to just minutes, contributing to the viability of the solution and ensuring Programa achieved return on investment. 

Furthermore, the solution has been successfully transitioned to Programa’s team and is working well. Programa’s own data and AI specialists now manage the deduplication system, ensuring it continues to provide value and adapt to future needs. Programa is now exploring additional applications of LLMs to address other challenges within the business.