Expert Briefing: Supervised vs. Unsupervised learning in AI
I had the opportunity the other week to spend a couple of hours shooting the breeze with our lead data scientist Tijl Carpels. While we chatted over an overpriced macchiato on everything from Star Wars to crypto currency the talk very quickly settled on the use of supervised and unsupervised algorithms in AI.
This all came about because we have launched a new bespoke AI product (ISAAC) and I really wanted to know more about how it worked and also why we chose this path.
“Let’s start with making a distinction, when I talk about algorithms, I’m talking about mathematical formulas used to solve a specific problem,” Tijl Carpels
The pure beauty of using AI to analyse customer data is that it can (if used right) find patterns and meanings that people would either:
Find it very hard to do
Find it impossible
Take too long to, rendering the analysis obsolete
So the theory is simple, but as many have experienced putting this into practice is fraught with dangers. Both from the side of the buyer and the creators of the technology. The problem a lot of us have is that in the mid 90’s many companies were sold ‘black box’ platforms that promised a lot and delivered little. With the rise of AI we end up in a similar situation with a distinct difference, when it’s done right AI does deliver.
Supervised vs. unsupervised learning
There’s a lot to be said on this subject, so I’ll try and not to geek out on you too much (I promise).
Wikipedia defines supervised learning as:
“Supervised learning is the machine learning task of inferring a function from labelled training data. The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal).”
I probed Tijl a little more at this stage, so why would a business choose this type of model, it seems expensive right?
“Supervised learning models are built for a specific business need, for example if you want to see how many customers are happy with the service they receive in your store.”
So now I have a lightbulb moment, if you want to find out what is going on in areas that are of interest to you this is the method to choose. In fact, it’s the method we have chosen for our primary AI ISAAC, built around our customers' needs not the other way around.
Supervised learning models have other great benefits too, for example if you wanted to train an algorithm to identify churning customers that is entirely possible.
Supervised learning however does present us with a few challenges, firstly you need to train the models on how to work. This requires human input, a great example of this would be Googles recapture. By using the power of the collective pool of users we are actually helping to train Google's AI in identifying everything from street signs to ocean vistas and more. Check out this article by Michael Lotowski to find out more.
Not all supervised learning models need this level of granularity to be good, our Algorithms regularly hit the 93% accuracy mark, which is pretty much as good as it gets and comparable to humans.
Wikipedia defines unsupervised learning as:
“Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labelled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data.”
"Unsupervised learning becomes useful when you have no labels, no gold standard in your data." Tijl Carpels
It does have its place, even with our own development pipeline we use it quite often. Just the other day one of our brilliant data scientists was presenting a topic modelling project she has been working on. When it’s done well it provides a wealth of knowledge that other systems can’t. However the problem is that most people either:
Don’t use it for the right purpose
Find it difficult to work with the results
I remember learning about the ‘black box’ solutions of the mid 90s / early 00’s, they were sold to us by data and CRM agencies as a product that just worked. Companies weren’t told how they worked or what they did ‘trust the system’ they would say. The problem is that a lot of these systems didn’t deliver the value they promised, it was too early for the technology and/or they were deployed badly. The simple fact of the matter is many organisations lack the granularity of data to be able to provide working models.
In order to make unsupervised learning work any dataset you need levels of volume and granularity in your datasets that most organisations don't possess.
Analysing text supervised learning gives more actionable insight
Unsupervised learning is great at spotting trends but the outputs can be confusing
A ‘one size fits all model’ really doesn’t work, go for something more customised
If you want to find out more on this subject please feel free to LinkIn with me or leave a comment below.