Machine Learning as an alternative to rule based processes

There’s a lot of discussion about machine learning these days and pretty much every one (vendors, users) is talking about it.

I remember attending courses on Artificial Intelligence, Machine Learning and even Artificial Neural Networks back in 1998. So what’s new?

How have AI and ML evolved?

I think a big reason why everyone is talking about machine learning now is that it it’s much simpler to use machine learning now for everyday, business use cases. Earlier, machine learning was mostly used for really complicated scenarios – think enterprise search (with advanced capabilities for proximity, sounds etc) or content analytics to do sentiment analysis. All these were useful but required expensive software and resources.

Not anymore. It’s become far easier to use machine learning for simpler problems. In fact, for lot of scenarios which required complex rules, you can actually use machine learning to take decisions. Let’s take an example. You are building a website that allows users to sell their old mobile phones. The website should be able to suggest a price based on a series of questions that a user answers. So you could have a set of rules that “rule-fy” each question.

For example:

Question 1: Phone model

If phone == A, Price = p

If phone == B, Price = q

Question 2: Age of phone

If phone == A, and bought within last year, price = P

If phone == A and bought more than one year ago but less than 2 years ago, price = 0.9 P

Question 3: Color

If phone == A, and bought within last year and color == black, price = P

If phone == A, and bought within last year and color == silver, price = 0.95 P

If phone == A and bought more than one year ago but less than 2 years ago, and color == black, price = 0.9 P

And so on. You can add more rules depending on questions about age, colour, defects, screen quality and so forth. And your rules become increasingly complex. And then what happens if a user wants to enter a value that the rule doesn’t handle?

Of course, in real life, you wouldn’t write rules like this. You will probably have a rules engine that that combines multiple rules and so forth but you get the idea.

Machine Learning as an alternative to Rules-based processing

Here’a how machine learning can replace a complex rules based application.

Let’s say you have historical data about phone sales. Yeah, I admit this is a big assumption but if you are creating rules and deciding prices, then you probably have some historical data anyways. So assume you have data such as this (this is just a sample; the more you have it, the better it is):

phone data

Fig: Second hand phone sales data

Now your original problem can be stated as a machine learning problem as follows:

How do you predict the price of a phone, that is not already there in the sample (or training set) above based on features and data available as part of training set?

Essentially, instead of you or your application making decisions based on pre-defined rules, you are now relying on your application to make decisions based on historical data. There are many techniques that can help you achieve this.

One relatively simpler technique is to use Linear regression. Linear regression is basically a statistical technique to predict an outcome (or dependent variable) based on one or more independent variables. Based on example above, you can describe Price P as a function of variables model, age, colour etc. Or in linear regression, it can be expressed as:

P = b0 + b1*model + b2*age + b3*colour + b4*condition…..

Machine learning algorithm then calculates values of b0, b1, b2 etc based on historical data and then you use this equation to predict price for an item that was not there in the training set. So if a new user now comes and offers a phone for sale on your site, you can recommend a price to her based on past sales.

Okay, that was a rather simplistic machine learning example and you can use many other more sophisticated techniques. For example, you can do a factor analysis or Principal Component Analysis (PCA) to reduce large number of items (e.g., news articles’ attributes) into smaller set of variables. Or use logistic regression instead of linear regression.. or whatever. The key point is that it is now much easier to use machine learning for everyday use cases without spending a lot on expensive software or resources. Pretty much all programing languages and development platforms have machine learning libraries or APIs that you can use to implement these algorithms.

The main drawback of using this approach (as in this example) is that the results might not always be as good as you would get with rules based technique. The quality of result is highly dependent on training set and as the training set improves (in terms of quality as well as quantity), the results would improve.

Are you using machine learning for your applications? If yes, what techniques are you using?

 

Some data from my social media survey

In my last post, I provided an overview of how to measure impact of a digital or social media initiative. But before you can measure the impact, you need to be able to create an inventory of what all activities in your organization can use these media.

In this post, let’s look at the findings of a survey I conducted in order to identify how organizations use social media.

First, let’s look at some profile data. There’s a lot more than what’s below, however, i’m sharing only a subset.

Profile of Participants

In terms of verticals or industry segments, the participants’ organizations were spread across all major industries. In fact, “All others” category, shown in the chart below, consisted of more than a quarter of participants and pretty much all industries were represented. However, bulk of participants were from service-oriented industries such as IT (21%), BFSI (8%), Consulting (11%) and Technology (10%).

organizations businessFig 1: Organizations’ primary business

About 37% of participants reported they worked for organizations that have more than 10,000 employees. Another 20% said their organizations employed between 500 to 9999 employees. These are representative of large organizations that typically have several complex processes and challenges.

no of emps

Fig 2: Total number of employees in organization

Usage Analysis

Participants were asked to select activities for which they could use social media. Results for activities that made maximum use of social media are shown below. This figure shows top 15 activities based on number of responses. Most marketing oriented activities figure in this list. That is not surprising. In addition, many use cases from other activities also figure in this list. In particular, social hiring (55%), customer support (53%), and knowledge management (52%) are amongst the most common activities for which social media is used.

social media in value chain

Fig 3: Social media usage in value chain activities: Top 15 activities

Another aspect of usage is in terms of social media tools. Users use these tools for any social media activity and so in addition to measuring usage of social media within activities, this is another way to measure usage. The figure below shows the percentage corresponding to usage of different social media tools. Social networking sites such as Facebook and Linkedin were at the top with 86% participants using them. Facebook’s usage for social media marketing and Linkedin for social hiring are well known use cases, so this result is not unexpected. This was followed by Blogs (64%), Video/Presentation sharing such as YouTube (59%), Instant messaging like WhatsApp (58%), Microblogs such as Twitter (56%) and Document and File Sharing such as Dropbox (55%).

social media tools usage

Fig 4: Social media tools usage

There’s lot more and i’ll share more findings in future posts. Meanwhile, i’m working on a set of tools that uses all this data as a basis for further analysis — something that organizations will be able to use for their own digital initiatives.

How do you Measure the Impact of a Social Media Initiative?

While there’s a widespread usage of digital and social media within organizations’ value chains, there are several questions that I hear often. E.g.,

  1. How  do you measure the impact of these initiatives?
  2. There are several use cases for which social media can be used. How do you actually prioritize these and decide what you want to use social media for?

Of course, there are many mechanisms – such as measuring financial impact (RoI etc) — to measure the impact of any initiative,  it’s not really trivial to apply these measures for something like social media.

One way to measure the impact or to be able to prioritize is by using some other outcome – a performance outcome or something similar.

 “Competitive Advantage” is one such outcome

A social media initiative can be analyzed in the context of whether or not you are you able to achieve competitive advantage by using social media. Again, there are many ways to define competitive advantage but for the purpose of this post, let us consider the concept of competitive advantage based on Michael Porter’s Value Chain Analysis (VCA).

There are numerous examples of social media usage in an organizations’s value chain activities. Some of these are shown in the figure below.

social media usage and its impactFigure: Social media usage and its impact on different sources of competitive advantage

The figure  above also provides an example of how an analysis of social media for competitive advantage could look like. For each activity in the value chain, the table shows examples of how social media can contribute to that activity and resulting impact on cost, differentiation and focus (the factors of competitive advantage, as described by Porter) for an example organization. Note that this figure only shows the summary of a sample analysis. In practice, an organization will typically use a detailed methodology and tools to access the cost, differentiation and focus impact of each of the activities and outcome will vary from one organization to another.

Once such an analysis is done, an organization can find out what are the most important activities in terms of their impact on different sources of competitive advantage. Based on organization’s own strategy and vision, appropriate activities can then be chosen as a use case for social media implementation.

In next few posts, i’ll go into details and describe how to analyze social media in context of competitive advantage. It’s all part of my PhD work in which I define several constructs (or factors) that have an impact on usage of social media and ultimately on competitive advantage. We’ll also explore how these different constructs play with each other, and finally look at a framework (and an excel-based tool) that can help you with any digital initiatives in your organizations.