In the last decade, artificial intelligence has accomplished many significant feats: from classifying an image to tracking objects in video feeds, from beating a Go player to beating a Starcraft 2 pro and more. After such a successful decade, it is evident that academic advance is slowing down and the research lately is more incremental than fundamental. However, that also means that the AI technology is maturing and we are finding new domains where technology can be applied to.
Having seen/ read/ been amazed by these applications as a business member you maybe be starting to wonder whether it is time for you to try out these approaches in your work. In this blog post, we will share our experience in solving business problems with AI and hopefully help you avoid repeating our mistakes.
Before jumping into the core matter, we should make a distinction between two flavors of deployment issues:
Problem 1: Adding ML to the IT stack. ML is compute-heavy and requires compute-oriented software and hardware which might not be present in IT infrastructure. Therefore, how and where the algorithms are run needs to be decided, and appropriate skills need to be added to or developed in your IT department.
Problem 2: Dealing with the implications of having ML in your business processes. Most of ML methods are non-deterministic and they are evaluated on a statistical basis. That means that even when the methods work it is possible for them to deliver the wrong result. This stochastic nature of the algorithm can be a novelty to a business which depends on a fully deterministic IT system. Business processes affected by such randomness need to be identified, the impact evaluated and, if needed, policies to address it are required.
Problem 1 heavily depends on the existing infrastructure and the algorithm specifics. Moreover, it is the more obvious of the issues and has been documented extensively. To the extent that it could be solved by completely outsourcing it to the cloud - store the data, train the models, use the models, all in the cloud. There are multiple good tools to achieve that, for example Google cloud offers AutoML and Amazon AWS offers multiple AI/ML services, just to name a few.
There is much less information on Problem 2, but it can make or break the business proposition of an AI/ML application. Usually, when clients approach AI/ML developers they ask “Can you solve problem X?”, developers reply “Sure, what accuracy do you need?”. The jump from a problem to an AI algorithm is a non-trivial one and requires careful attention.
To link the business problem to an ML algorithm one needs to deeply understand where the value is added in the process. For example, AI is increasingly being applied for initial screening of medical patients and there you should be extremely cautious about false negatives, i.e. where the patient is screened as healthy when in fact he might be severely ill. The impact of the false positive in that scenario would most likely only lead to and increase in costs. Ignoring these aspects of accuracy could lead to a deadly outcome!
Therefore, framing the real-life problem into an AI solution is a significant part of the development and in this blog series, we will share our experience and tips on how to avoid some of the dangerous pitfalls.
We start by sharing two examples that we’ve encountered that are spaced nearly a decade apart. The older one will be used to set a reference point of what developers and IT managers are used to and the more recent one will shed some light on how AI algorithms are developed nowadays. Contrasting the two allows us to compare in what ways we should change how we think about development and how to be prepared to manage what is to come.
Although we focus on image recognition, the issues are shared across a wide variety of data-based ML approaches.
In 2012 we were approached by a client to develop licence plate recognition from images for their parking lot. The client gave us a dataset of 100 images and we started to work on it.
Recall that in 2012 the situation with respect to AI/ML was very different. Although deep learning had been making waves in the research community, there were much fewer software tools available, much less knowledge on how to train and use neural nets, and it was much harder to run algorithms on GPUs.
The way production computer vision was at the time - use heuristics to simplify the problem and sometimes use some neural network approach to solve that simple problem.
The solution we came up worked along these lines:
Select the regions fulfilling certain conditions:
Loop over the selected regions and
For each of the letter-like contour try to extract the letter by either:
The solution took us a few months and was fairly restrictive:
In many cases, the algorithm crashed at one of the steps and did not return any result. In the cases it worked, the accuracy reached a respectable 95%. The overall accuracy was fairly low - maybe around 60% since in many cases it did not return, but given that cameras do 30 frames per second the overall solution was worth the effort. The problem was solved and has been used in production for a while.
More recently in 2019, we were approached by a car-wash network manager who wanted to measure vehicle queue length using cameras.
We received a sample of 1000 images and started developing the solution. After manually labeling, we developed multiple candidates, but the high-level view of the solutions was:
The development of the principal algorithm took us a few days and it was 96% accurate. However, even though we had good accuracy in our sample which the client would accept as appropriate, we did not stop there and we’ll share why later on.
Let’s compare the two approaches:
Heuristic-based algorithmML-based algorithmRequires less dataRequires more dataThe code is complicatedThe code is simpleThe decision logic is obviousA lot of maintenanceLikely to crash than give a wrong resultNever crashes, but might give the wrong answerSlow developmentFast developmentRuns on any PCFor efficient use requires GPU-enabled PC
These contrasts led us to identify the following challenges when working with AI algorithms.
In this blog post, we shared our experience with two similar problems and their two different solutions. The heuristic approach has been in use for a long period of time and the AI one is slowly taking over. Contrasting how the two are developed and used hints at the challenges of the new approach. Having figured out that in this blog post we will elaborate on how such projects should be structured to achieve maximum success in the next part.
In the last decade, artificial intelligence has accomplished many significant feats: from classifying an image to tracking objects in video feeds, from beating a Go player to beating a Starcraft 2 pro and more. After such a successful decade, it is evident that academic advance is slowing down and the research lately is more incremental than fundamental. However, that also means that the AI technology is maturing and we are finding new domains where technology can be applied to.
Having seen/ read/ been amazed by these applications as a business member you maybe be starting to wonder whether it is time for you to try out these approaches in your work. In this blog post, we will share our experience in solving business problems with AI and hopefully help you avoid repeating our mistakes.
Before jumping into the core matter, we should make a distinction between two flavors of deployment issues:
Problem 1: Adding ML to the IT stack. ML is compute-heavy and requires compute-oriented software and hardware which might not be present in IT infrastructure. Therefore, how and where the algorithms are run needs to be decided, and appropriate skills need to be added to or developed in your IT department.
Problem 2: Dealing with the implications of having ML in your business processes. Most of ML methods are non-deterministic and they are evaluated on a statistical basis. That means that even when the methods work it is possible for them to deliver the wrong result. This stochastic nature of the algorithm can be a novelty to a business which depends on a fully deterministic IT system. Business processes affected by such randomness need to be identified, the impact evaluated and, if needed, policies to address it are required.
Problem 1 heavily depends on the existing infrastructure and the algorithm specifics. Moreover, it is the more obvious of the issues and has been documented extensively. To the extent that it could be solved by completely outsourcing it to the cloud - store the data, train the models, use the models, all in the cloud. There are multiple good tools to achieve that, for example Google cloud offers AutoML and Amazon AWS offers multiple AI/ML services, just to name a few.
There is much less information on Problem 2, but it can make or break the business proposition of an AI/ML application. Usually, when clients approach AI/ML developers they ask “Can you solve problem X?”, developers reply “Sure, what accuracy do you need?”. The jump from a problem to an AI algorithm is a non-trivial one and requires careful attention.
To link the business problem to an ML algorithm one needs to deeply understand where the value is added in the process. For example, AI is increasingly being applied for initial screening of medical patients and there you should be extremely cautious about false negatives, i.e. where the patient is screened as healthy when in fact he might be severely ill. The impact of the false positive in that scenario would most likely only lead to and increase in costs. Ignoring these aspects of accuracy could lead to a deadly outcome!
Therefore, framing the real-life problem into an AI solution is a significant part of the development and in this blog series, we will share our experience and tips on how to avoid some of the dangerous pitfalls.
We start by sharing two examples that we’ve encountered that are spaced nearly a decade apart. The older one will be used to set a reference point of what developers and IT managers are used to and the more recent one will shed some light on how AI algorithms are developed nowadays. Contrasting the two allows us to compare in what ways we should change how we think about development and how to be prepared to manage what is to come.
Although we focus on image recognition, the issues are shared across a wide variety of data-based ML approaches.
In 2012 we were approached by a client to develop licence plate recognition from images for their parking lot. The client gave us a dataset of 100 images and we started to work on it.
Recall that in 2012 the situation with respect to AI/ML was very different. Although deep learning had been making waves in the research community, there were much fewer software tools available, much less knowledge on how to train and use neural nets, and it was much harder to run algorithms on GPUs.
The way production computer vision was at the time - use heuristics to simplify the problem and sometimes use some neural network approach to solve that simple problem.
The solution we came up worked along these lines:
Select the regions fulfilling certain conditions:
Loop over the selected regions and
For each of the letter-like contour try to extract the letter by either:
The solution took us a few months and was fairly restrictive:
In many cases, the algorithm crashed at one of the steps and did not return any result. In the cases it worked, the accuracy reached a respectable 95%. The overall accuracy was fairly low - maybe around 60% since in many cases it did not return, but given that cameras do 30 frames per second the overall solution was worth the effort. The problem was solved and has been used in production for a while.
More recently in 2019, we were approached by a car-wash network manager who wanted to measure vehicle queue length using cameras.
We received a sample of 1000 images and started developing the solution. After manually labeling, we developed multiple candidates, but the high-level view of the solutions was:
The development of the principal algorithm took us a few days and it was 96% accurate. However, even though we had good accuracy in our sample which the client would accept as appropriate, we did not stop there and we’ll share why later on.
Let’s compare the two approaches:
Heuristic-based algorithmML-based algorithmRequires less dataRequires more dataThe code is complicatedThe code is simpleThe decision logic is obviousA lot of maintenanceLikely to crash than give a wrong resultNever crashes, but might give the wrong answerSlow developmentFast developmentRuns on any PCFor efficient use requires GPU-enabled PC
These contrasts led us to identify the following challenges when working with AI algorithms.
In this blog post, we shared our experience with two similar problems and their two different solutions. The heuristic approach has been in use for a long period of time and the AI one is slowly taking over. Contrasting how the two are developed and used hints at the challenges of the new approach. Having figured out that in this blog post we will elaborate on how such projects should be structured to achieve maximum success in the next part.