How Experience Design and Machine Learning can join forces to create powerful new products that exceed customer expectations to drive growth, revenue, retention and advocacy.
How ML is being used in products already
AI and ML are already being used to enhance products and experiences around us every day. A lot of it happens in the back end – measuring, capturing, computing and learning. Every interaction you take online as a consumer can help ML become much more effective and relevant.
Even your smartphone uses every interaction to build its own model around you and your behaviour, and can predict what you need based on your habits. If you try searching for an app, you’ll notice it starts by recommending your most frequently used ones. Online streaming services like Netflix also use AI to recommend shows and movies based on algorithms that track the types of shows you watch, for how long, what you rewatch and also comparing viewing habits of people within your demographic.
They say that “data is the new oil”, which is a great way to understand how critical it is for ML. Data is the fuel that powers the algorithm for ML models to generate results. And generally speaking, the more data you have, the better your algorithm will be. Even more importantly, your ML model will only be as good as the data it gets. Like food for humans and soil for plants, without data, your AI will not thrive. With the right food, the right soil, your AI will learn faster and better.
The ultimate goal of utilising ML in product development is creating a virtuous success cycle based on gaining better recommendations to build a better product to drive more usage, which provides more data for training which, in turn, creates further recommendations.
How to consider ML in product development
There are three main ways ML can be used in your product development:
- Improve existing features e.g. using recommendation engines to personalise suggestions for an existing product.
- Enable new features e.g. unlocking your phone with facial recognition instead of a passcode.
- Enable new products e.g. creating a new product around ML like bolttech’s Click-to-Protect, a computer vision-based screen diagnostic that helps bolttech to understand device health when offering insurance protection for a wide range of devices.
Let’s drill further by using the ‘pirate metrics’ model which iIllustrates the problems that products must resolve throughout a customers journey, and some common examples of where ML is already being used to help.
- Acquisition
The problem? Users aren’t aware of your product, so how do you build awareness within your target audience.
How ML can help? Digital advertising and promotion that uses targeted methods to reach specific consumers. - Activation
The problem? How to onboard your customers when they’re interested in purchasing your product or services so they start off on the right foot.
How ML can help? Personalised onboarding process that ensures users can join simply and easily. - Retention
The problem? Retention of the customer and their continued use of your product.
How ML can help? Prediction, enhancement or recommendations. - Referral
The problem? To the delight or value of your product to be referred to friends or family.
How ML can help? Holistic experiences. - Revenue and profit
The problem? Revenue and reducing cost to increase revenue/profit.
How ML can help? Automation.
Your business goal may touch on one or more of these, and the solution will be slightly different for each.
Working out your ML objective
Working out what you want ML to achieve is the most critical part of this process and the hardest thing to get right. You need to work out what you want the algorithm to do and what data you need to get the results.
It’s best to start with your overall business goal and what problem you’re trying to solve for end users. Once you have brainstormed different ideas and solutions, you can work through the ML objective. Your objective will start to dictate the next steps, such as your model selection, your data engineering and the type of answer you’re looking for. An oversimplified view of what the objective should be could lead your algorithm down the wrong path. Use your intuition first – then experiment through prototyping and iteratively improving your result.
How to develop an ML solution
From an ML perspective, depending on the dataset, the problem at hand, and the time available, there are a few different options for creating ML models.
- MLaaS are the off-the-shelf tools available to us by cloud providers. These services are usually very easy to use but are generic so might not perform well enough for what you need. However, using them requires very little knowledge of ML so they’re a good way to start.
- The next level is AutoML services which are available as open source tools or on the cloud. These require some knowledge of ML but take away the model design from the user. For these, you pass your dataset or inputs/features to your AutoML tool, and it trains a model on your data. For such services we require an average amount of effort but the performance and accuracy can be improved if you involve a ML Specialist who can implement a custom model.
- In a custom model option, you need ML expertise, but this allows you to apply state of the art research and ML models to your problems. This obviously requires more effort but the highest performance is expected.
Bringing an ML product to life
Once you’ve determined your business goal and ML objective, the next step is to bring it to life. There are four key stages that a combined XD and ML team can work through together.
1. Understand & prioritise
The first stage is to understand both the business requirements and how ML can help. It’s important to prioritise ideas by business value and complexity, such as whether you have the right data needed, the technical risks involved and how ML can be implemented.
2. Experiment & iterate
It’s best to start small, focusing on one or two specific ML objectives, then agree on performance objectives (such as success metrics, model accuracy). By experimenting through a Proof of Concept (PoC), you can confirm whether all the data you need is available, how to prepare the data, demonstrate technical feasibility and evaluate the performance of your ML Model. The focus of a POC is to implement the model and find what works and what doesn’t.
You can do this by:
- Defining the right metrics that measure the success.
- Developing an iterative process to train the right model.
- Defining a time-boxed engagement.
- Analysing the data and coming up with the right features and inputs to the model.
- Choosing the right model that fits the data and is suitable for the problem at hand.
- Training the model on the dataset.
Once you’ve implemented the POC, the next step is to evaluate and iterate. In different iterations you can either:
- Iterate based on a model-based approach: focus on the model and try to change the model in such a way that a better performance is a achieved (this is within the time-boxed effort required for a PoC), or
- Iterate based on a data-centric approach: where you consider the model to be fixed, look closely into the model predictions, and try to collect or generate more data that covers the cases where the model makes mistakes. Since this might require more data collection, it is a longer term process, and therefore, might go beyond a time-boxed PoC.
3. Build & launch
Once you have tested the approach through a POC, you can then move into production. This includes developing the software application or integration, the data infrastructure and pipelines, model deployment and scale. This will enable you to launch a Minimal Viable Product (MVP) to get started.
There are different dimensions that need to be taken into account when moving from POC to production:
- Choosing the right infrastructure, which allows your model to be responsive with reasonable latency and throughput, and is not very expensive to run.
- Trying to automate the model retraining gradually. This allows the data science members of the team to work on the model improvement or retrain end-to-end with the least amount of friction.
- Implement data pipelines so that an end-to-end training or retraining of the model is possible. After this, you’ll have CI/CD pipelines in place so that any change in code or data will automatically trigger the whole pipeline, train the model, evaluate, and in case a better model is achieved, replace the model in production.
- Gradual rollout. You need to gain confidence that the ML model works well in practice. Therefore, try to roll out the model gradually. In this process, you might start from a fully-manual pipeline where a person makes the judgement call on the data, and compare the human judgement with the model prediction. As you gain more confidence in the model, you can send less and less data to the human and rely more and more on the model.
- Once you’re confident with the model, remove the human intervention from the pipeline and fully replace it with the model. You still need to implement monitoring on the model to make sure it stays performant and no data or model drift would take place as time goes by.
In ML applications, some risks might arise because the model might be biased due to the dataset, or due to the fact that the model might not give an accurate prediction and how damaging a wrong prediction might be. For instance, in applications such as predicting the credit-worthiness of a client applying for a loan, or driverless cars, a mistake could have a massive impact.
The solution to this problem is to start off with a human in the loop pipeline. In this process, an example is passed to the model, and in scenarios where the model is not very confident about its prediction, the data is directed to a person who would make their judgement on the case. In this process, the aim is to de-risk the consequences of a bad judgement by the model. As another advantage of this pipeline, a more comprehensive dataset will be gradually collected, especially in scenarios where the model lacks a good judgement. This data is very valuable and can be used for retraining the model.
This loop continues in practice as the model will learn to correct itself in wrong or unconfident cases, and you can decrease the engagement of human intervention. After the model reaches an acceptable maturity, you can move to automatic monitoring of the model and automatic health checking and triggering for manual review. This process is time-consuming but is the right approach to reach a reliable ML, and hence brings you closer to a fully-automated feature backed by ML models.
4. Monitor & Improve
Once an MVP model is deployed, you can then monitor performance, retrain with new data, manage versions and re-deploy. It’s a constant cycle.
An example using a DiUS client: bolttech
bolttech is an international InsurTech with a mission to build the world’s leading technology-enabled ecosystem for device protection and insurance. With a full suite of digital and data-driven capabilities, bolttech makes it easier and more efficient to buy and sell insurance and protection products. Using the business goal and ML objective below, two ML models were applied:
Model 1: Real-time phone detection
- Assist user with correct positioning of the phone in front of the mirror
- Optimised to run on the smartphone (edge model)
Model 2: Screen classification
- Categorising as cracked or not-cracked
- Larger more complex model hosted on the cloud
- A combination of instructions and visual cues to help guide the user. ML allows for real-time phone tracking while users are aligning their phones
- Recognising a phone is present
- Positioning the phone in the right spot for the right time at the right angle
- Removing as much interference as possible
- A red bounding box indicates that phone is either too far away or too close and the green box indicates the position is good with additional instruction to hold the phone still while the capture is taken
- Introducing an auto capture also meant we could remove the need to tap the screen to take the photo.
You can find out more about the bolttech story here.
We can help
For more help or support with your product development journey, don’t hesitate to contact DiUS. We can take stock of your current circumstances and situation before advising on the next steps.
Complimentary two-hour Machine Learning workshop
We can help you through all the stages of this journey and would like to offer a complimentary two-hour ML workshop. We’ll help you get started with Machine Learning or review and recommend strategies to improve an existing ML-powered solution.
Benefits of our ML workshop include:
- Practical recommendations to improve machine learning performance, quality and reliability.
- Expertise across the machine learning product lifecycle and solutions from MLaaS and
- AutoML to custom state-of-the-art models.
Holistic approach that considers data-centric and model improvements.