Machine Learning (ML) algorithms, without a question, have improved our lives by providing us highly useful products in every possible domain or field. In every case or product, the prediction is not always 100% accurate or doesn't need to be.
Should you continue improving the accuracy of your machine learning model?
Know when to stop improving your models to maximize product value
Machine Learning (ML) algorithms, without a question, have improved our lives by providing us highly useful products in every possible domain or field. In every case or product, the prediction is not always 100% accurate or doesn’t need to be. It would be extremely challenging or impossible to achieve near 100% prediction accuracy. So what is a good stopping point or when we should stop improving an algorithm?
To achieve a good ML model — there are lot of well-known techniques such as data cleaning (most important), regularization, data engineering, dimensionality reduction, feature selection, etc. It is possible to build a good ML algorithm with 80%–85% of accuracy using the above mentioned techniques however to achieve a better accuracy (85%–95%) it takes significant amount of time, effort, deeper domain knowledge, extreme data engineering, more data collection, and so on. In many cases it may not be worth the time or effort as it would have two major implications — 1. delay in product release (a competitor could take a lead and gain market leadership), and 2. increased development cost. We could always release an algorithm with good accuracy (75%–85%) and improvise it later — one and only if required. Please note that I am not talking about bug fixing and handling of some edge cases which is a part of the model maintenance routine. For certain domains or products, in fact it is actually critical to achieve >95% accuracy such as autonomous driving system, facial recognition, etc., and for such cases it important to keep improving the prediction accuracy.
Typically, development of any ML algorithm (or the product) follows the trend below for the model (prediction) accuracy.
Initially the leap region where the accuracy simply jumps to > 50% in case of classification algorithms with no or little effort in data cleaning and feature selection. Similarly for the case of regression algorithms in which case model accuracy is replaced by a score that falls between 0 and max based on the metric used.
region where the accuracy simply jumps to > 50% in case of classification algorithms with no or little effort in data cleaning and feature selection. Similarly for the case of regression algorithms in which case model accuracy is replaced by a score that falls between 0 and max based on the metric used. The leap region is followed by a linear region where for every unit of effort or time spent the algorithms get improved in similar proportion. A simple linear trend between model accuracy and cost or effort or time.
region where for every unit of effort or time spent the algorithms get improved in similar proportion. A simple linear trend between model accuracy and cost or effort or time. The linear region is followed by an exponential (model accuracy prospective) or asymptotic (cost prospective) region where as the model accuracy gets higher, more and more efforts (exponential) are required to achieve the same proportion of increase in accuracy. For example —Let’s say for the improvement that falls in the linear region, 70% –-> 75% we spend 2 units of time or cost. Then for the improvements in the exponential / asymptotic region — for 85% –> 90% we may need to spend 10 units of time or cost and to improve from 90% –-> 95% we may need to spend 100–1000 units of time or cost.
Why is 7 no lucky?
Lucky number 7 is even the basis for many myths and folklore. Ancient beliefs from around the world believed that the seventh son of the seventh...
To improve an algorithm beyond 90% range (depends on the application or product) that falls in exponential region we may need to recruit experts in the field (data scientists, Dev Ops, product managers, …), provide expensive compute nodes and buy or subscribe more tools and software required, involve in extensive research, etc., which would significant increase the cost of the algorithm building and maintenance. For some products it makes sense to spend significant amount of money to improve the accuracy and performance to outrun the competition (autonomous driving system or stock price forecasting) whereas for some consumer products (item recommendations for shopping and classification of images) it may not make sense.
“Product value is the benefit that a customer gets by using a product to satisfy their needs, minus associated costs. Complexity is the effort associated with delivering such a product to the customer.” - Hubert Palan
Product Value = Benefit — Cost
The additional cost spent for improving the accuracy of the models directly impacts the product value. Although the benefit increase, the cost would increase in multiple folds as we push the accuracy higher and higher. In my opinion, the different trends between the product value and the model accuracy would look like as depicted in the sketch below. The trends depend on various factors such as the constituent domain or field, product type, demand, competition, etc.
Exponential Trend: The value of the products such as autonomous driving systems, stock forecasting, text recommendation systems, etc., increases exponentially with increase in the model prediction accuracy. No one would be interested in driving an autonomous vehicle with lot of FPs and FNs which makes it totally unreliable and poses significant risk to lives on board. Similar, text recommendations will be skipped or ignored if not accurate and stock forecasting will be ignored if the predicted range is broad. As we push the algorithm to yield accurate results, the product gains customers and would become a market leader.
The value of the products such as autonomous driving systems, stock forecasting, text recommendation systems, etc., increases exponentially with increase in the model prediction accuracy. No one would be interested in driving an autonomous vehicle with lot of FPs and FNs which makes it totally unreliable and poses significant risk to lives on board. Similar, text recommendations will be skipped or ignored if not accurate and stock forecasting will be ignored if the predicted range is broad. As we push the algorithm to yield accurate results, the product gains customers and would become a market leader. Asymptotic Trend: For other type of products such as energy forecasting, demand forecasting, product recommendations, movie recommendations, image classification, etc., the product value increase as we improve the accuracy of the algorithm — even with 60–80% accuracy people would still be interested in using the product as compared to the other set of products that we discussed earlier that follows exponential trend.
What sport is considered the hardest?
According to Sports Virsa, the top 10 hardest sports in the world to play in 2022 are as follows: Boxing (hardest), American football, mixed...
For other type of products such as energy forecasting, demand forecasting, product recommendations, movie recommendations, image classification, etc., the product value increase as we improve the accuracy of the algorithm — even with 60–80% accuracy people would still be interested in using the product as compared to the other set of products that we discussed earlier that follows exponential trend. Others: There could be definitely other trends depending on the factors mentioned above but these two would be the commonly observed trends between the product value and the model accuracy / cost / time / effort.
If you are going to develop algorithms by accounting in all these factors (product value, cost, accuracy, time, and effort), you will definitely build highly valuable products in short time.
Thanks for reading! I hope this was helpful. Please leave your feedback and help in further improving the content.
Wager Mage
Photo: Tima Miroshnichenko
Promotion
Promotion
Promotion
Promotion