Exploring the world of multilingual chatbots with generative AI

I recently found myself at the centre of a flurry of questions about multilingual chatbots. DiUS was approached by two clients with similar challenges but from different industries. One client, a healthcare provider specialising in a highly stressful treatment, needed a way to communicate effectively with its diverse clientele, many of whom do not speak English. Clients need their questions answered in their native languages to feel understood and supported. The other, in the travel industry, was seeking a solution to provide the best experience to its myriad of non-English speaking visitors. Both scenarios underscored a common need: breaking down language barriers to improve user experience.

This is where multilingual chatbots powered by generative AI come into play. These chatbots promise seamless communication for users from around the globe. Let’s dive into multilingual chatbots, our investigative journey, the outcomes, risks, and our recommendations for those considering this technology.

Investigation process

Our investigation into multilingual chatbots aimed to determine whether these were viable solutions for our clients. Our goal was to gather enough information to make a go/no-go decision on a deeper discovery process. We had two primary objectives:

1. Assessing the competence of large language models: We wanted to evaluate how well these models could handle non-English queries. We chose Mandarin as an example language for this assessment.
2. Testing different strategies: We explored various approaches to improve the models’ responses, including direct query translation and answering queries in English before translating them back.

Setting up the investigation

We used OpenAI’s GPT-4 for this investigation. Here’s a breakdown of our approach:

• Scenario-based queries: We created realistic scenarios that users might encounter. For example, for the travel industry client, queries included “Where is the nearest restroom?” and “How far away is the Opera House?” These queries were crafted in both English and Mandarin.
• Translation approaches:
  • Direct query translation: Passing the query in Mandarin directly to the language model and getting the response in Mandarin.
  • Intermediate translation: Translating the Mandarin query to English, generating the response in English, and then translating the response back to Mandarin.
• Evaluation metrics:
  • Translation accuracy: How accurately the model translated queries and responses between languages.
  • Relevance and correctness: How relevant and accurate the responses were to the queries.
  • Level of detail: The depth and helpfulness of the responses.

Technical details of the models

We utilised OpenAI’s GPT-4o model, known for its robust natural language processing capabilities. The model is trained on a diverse corpus of multilingual data, including a significant amount of English and Mandarin text. This extensive training allows the model to handle a wide range of topics and language nuances. For translation tasks, GPT-4 leverages its neural machine translation (NMT) capabilities, which enable it to understand context and provide more accurate translations compared to traditional phrase-based translation methods.

Results of the Investigation

Our findings were promising:

• Translation accuracy: The models performed well in translating queries and responses between English and Mandarin, maintaining a high level of accuracy.
• Relevance and detail: Both approaches yielded relevant and detailed responses, but translating the query to English before generating an answer generally produced more detailed results. This is likely due to the model’s stronger performance with English text.

Specifically, the intermediate translation approach (translating queries to English first) resulted in:

• Higher relevance: The responses were more contextually accurate, likely because the model’s training data had more comprehensive coverage in English.
• Better detail: Responses included more nuanced information, providing a richer user experience.

Risks associated with multilingual chatbots

While the potential of multilingual chatbots is exciting, there are inherent risks:

• Increased vulnerability to jailbreaks: Multilingual models are more susceptible to being manipulated or confused by complex queries in different languages.
• Guardrail challenges: Ensuring that safety and control mechanisms work across multiple languages is complex. For example, content moderation filters and bias mitigation strategies must be effective in all supported languages. This requires extensive testing and fine-tuning.
• Consistency in user experience: Maintaining a consistent user experience across languages can be challenging. Different languages might require different conversational designs, complicating development and maintenance.

Should you implement a multilingual chatbot?

This is a common question, and the pros and cons should be carefully evaluated. Multilingual chatbots are particularly beneficial in situations where the alternative is a complete communication breakdown. If a non-English-speaking user cannot interact with your system at all, a multilingual chatbot, despite its imperfections, offers significant business value by enabling some level of interaction and support.

Key points to consider:

• Business value: If your clients include significant numbers of non-English speakers, the ability to communicate in their native languages can enhance customer satisfaction and engagement.
• Alternatives: Compare the chatbot’s performance not to a perfect English-language interaction, but to the scenario where users have no assistance at all in their language. The relative improvement can be substantial.
• Contextual understanding: Ensure that the chatbot can handle the specific context and queries relevant to your business, which might require customization and continuous learning.

Recommendations for implementing multilingual chatbots

For organisations considering multilingual chatbots, here are our recommendations:

1. Make the chatbot work in english. It makes sense to begin with the training data that has the most comprehensive coverage. Instead of allowing free-form inputs, guide users to select their language at the outset. This helps maintain control over the chatbot’s performance and reduces the risk of errors.
2. Start with a target language: Begin by implementing chatbots in the most critical non-English language for your users. This allows for focused development and testing.
3. Continuous testing and improvement: Regularly test and refine the chatbot with real user interactions in different languages. This iterative process ensures the chatbot evolves and improves over time.
4. Consider the entire user experience: Think beyond the chatbot. Consider how multilingual support fits into the broader user experience. This might involve integrating the chatbot with other services and ensuring consistency in communication across all touchpoints.
5. Enhanced guardrails and moderation: Implement robust guardrails to prevent misuse and ensure safe interactions across languages. This includes developing language-specific content moderation and bias detection mechanisms.

It’s also important to consider that the generative AI space is constantly evolving, so bake in effort to continuously iterate with new and improved large language models.

Conclusion

While there are challenges, the benefits of multilingual chatbots are undeniable. They open up new avenues for customer engagement and support, making services more accessible and user-friendly. With careful implementation and ongoing refinement, multilingual chatbots can be a valuable asset for any global organisation.