We've touched on this already to some degree, but is there reason for us to believe that social media sentiment is going to align with public opinion? Well, we've already seen an example where that may not be the case, just based on the volume numbers. Same problem's going to carry over to a social media sentiment. That is, the population of interest. When we conduct traditional marketing research, where we construct a sample that is representative of our population, let's say our population that we're interested in is our customer base. Right, well, we make sure that we get a representative sample from that customer base. Whereas, if I'm looking at the sentiment of social media, I might only have my hardcore supporters or my hardcore detractors who are contributing comments. They're going to give me positive comments, they're going to give me negative comments. The big group that I'm missing out on are the customers who are positive enough that they buy our product, but they don't have a strong enough connection to the brand that they're contributing to social media. That's a place where we might expect to see a disconnect between the sentiment expressed in social media and what we would collect through traditional marketing research studies. If we see a low sentiment on social media, should we immediately worry? Well, not necessarily because we've gotta understand who's contributing comments that are feeding into that low sentiment score, and same thing if we see a very high sentiment. We only might be hearing from a subset of the population of interest, and if we make decisions based on that subset, it may end up alienating the broader population. Another concern that we've gotta keep in mind when we're doing analysis, especially about brands, are we talking about sentiment surrounding the brand? Or, are we talking about sentiment specific to a particular product or an attribute of a particular product? And so if we take Apple as our brand well, they've got a lot of products that people might be talking about. Well each of those products also, we've got hardware, we've got software, and even within those software, if we're looking at let's say people talking about the iPhone. Well we've got apps that are on the iPhone. So we need to be able to understand Is the sentiment about the overall brand? Is the sentiment about a particular product? Is the sentiment about a particular feature. We're getting better at being able to do this. If you use manual coding, of course we can describe the sentiment expressed in statement to a particular brand or product or feature. Even with automated approaches, if we, if we're using natural language processing, we can try to ascribe the sentiment toward the, the specific product or the specific attribute. But. If we were to just look at all comments mentioning Apple as the brand. And let's look at that sentiment. Some of those mentions are going to be about iPhone, some of those are going to be about the iPad, some of them are going to be about computers. Some of them were going to be talking about software. Some of them were going to be talking about apps. Some of them were going to be talking about price. Well, there's reason to expect variation and sentiment across products or cross the attributes that are being discussed. And that's something that we're going to have to keep in mind when we're doing our analysis using social media data So we've talked a little bit about the metrics that we can report and I wanted to use this opportunity to illustrate one of the problems that we've got with composite scores. Let's say that we're looking at the distribution of comments coming in on a continuous scale, all the way from extremely negative to extremely positive. And maybe it looks like a velcro, maybe it looks like a normal distribution. And this sentiment is somewhere in the middle. So we could say okay, let's call that On average it's neutral right. The positives and negatives wash each other out so we say people are neutral. But we could have a very different distribution. In this case we've got a bimodal distribution Equal number of positive and negative comments, again, it's going to have the same average. And so two very different populations on, in one graph what we're seeing is most people are in the middle. On the other graph, people are on one of the two extremes. But when we just look at the average, we are going to lose that information. It's also worth mentioning, perhaps that, When we talk about neutral comments, there should be a distinction made between comments that are truly neutral, that is, I don't have an opinion being expressed, versus comments that are mixed in terms of the sentiment that's being expressed. They express both positive And negative aspects. Those don't necessarily cancel each other out, so we probably want to take a little bit more care when we're talking about comments and look at positive comments, neutral comments, negative comments, as well as mixed comments. All right We've talked a little bit about variation and sentiment that can be driven by the product or by the attribute that's being discussed. Another driver of variation and sentiment that we found is the venue format that you're pulling your social media data from. And so, these are graphs taken out of a study that we had done Using data from a B2B telecom provider. We had comments pulled from blogs, discussion forums, and micro blogs, the largest micro blog being Twitter. And what we've plotted out here, are the fraction of comments over time, that were positive Neutral and negative. And you can see that within the positive comments, within each of these we see differences across the blogs, discussion forums, and microblogs. And so for example, among the negative comments, they're most prominent among the discussion forums whereas Blogs tend to have more positive comments. So, if we average all of these together, then we're losing some of that information about why we might be seeing that variation. There may be structural reasons for why blogs are more positive, for why Discussion forms are more negative. So we want to take that into account whenever we're doing analysis using this social media data. So how can we use the information that Social media listening platforms are going to provide. Sentiment and volume are two of the most popular metrics but I would argue that you don't want to look at them in isolation of each other. So what would happen if we were to try to look at those two measures In combination so I've put together a two by two. It might be a little more coarse, but I think it gets the idea across. So let's say we have row volume and row sentiment. All right, well, we'd rather have higher sentiment so it's not ideal from where we stand But we also have low volume this is something that is not being discussed to a high degree. This is a little bit this is something that I consider a nuisance. I am not saying ignore it completely but the low volume of conversation is telling us that it has not reached that critical degree yet when we might need to take action If we were seeing positive sentiment around a particular topic and still low volume, well it's better than seeing the negative sentiment. But, again, the volume isn't high enough for us to declare this a success. So, there's the [INAUDIBLE] here there is something that might start to grow and we should be paying attention to, we should [INAUDIBLE], but right now, it's just in its early stages. There is potential when I can be ready to declare that a success. The red flags, those who are going to be the one where the sentiments is lower than you're expecting it to be and the volume is higher than you're expecting it to be. And, This is not going to come out of the middle of nowhere. We probably are going to see early signs of this. So, if we're able to engage in social media monitoring early on, something that might look like a nuisance, gradually. slowly as it picks up steam gets that higher volume that's when we are going to start to worry about it. Once it's hit that higher volume that's when we have to take action the hope would be through active monitoring. To make that determination as early as possible and be in a position to deploy resources accordingly. Of course, if the sentiment is positive and we're seeing more and more volume pick it up. These are those viral success stories that you hear about. And so to give you an example of kind of a company reacting to a red flag Kraft foods manufacturer of Oreos they were sued by a consumer advocacy group about the use trans fat in 2003. A study was conducted by BuzzMetrics looking at more than 2 million comments mentioning trans fat And those comments came from more than 100,000 people. So it's a decent volume of conversation. The part that stuck out here, though, is Kraft is mentioned in almost 20% of those comments. And in more than 25% of those comments, Oreos. AR are mentioned for their inclusion of transfer. Well, within two months, Kraft announced that it was going to begin removing transfer from all of their products. And so, I'm not saying that necessarily this particular study, or this particular law suit Is what triggered Kraft to make determination to remove from their products. But if we were engaging in social media monitoring all along, and we see the volume of conversation, who are the contributors, To this conversation. Alright to get 2.6 million comments from just over 100,00 people on average, that's averaging more than twenty comments per person. So this a very involved group. This is not, you know, one person contributes one comment. People contributing more than twenty comments based on this topic These are people who are invested in it, they maybe influential within the online community, they maybe influentials within an offline community. So that's something that we want to is how invested are the contributors in this particular topic? How influential are these particular contributors? But, may have started out as a nuisance issue for Kraft As we start to see those metrics evolve over time, definitely a red flag and you would want to take action based on that. So just a couple of caveats to keep in mind, I've already mentioned both with regards to volume and sentiment, you want to establish what is normal, what is the appropriate baseline. If you're McDonald's, and you're expecting 90% of your comments on social media to be positive, it's not going to happen. You need to look at what's normal for our brand. What's normal for our particular industry, and set your base line according Right. As far as sentiment, there has been a lot of discussion around the use of automated sentiment tools. One of the problems with automated sentiment tools is that most comments are included as neutral. All right, if If I can't recognized something as clearly positive as clearly negative. It may end up getting coded as neutral. Neutral may also be confounded with mix comments that have both positive and negative sentiments. You'll see a lot of statements made that our programs have 80 to 90% accuracy. We'll keep in mind that that 80 to 90% accuracy includes neutral comments. So if I were to take all comments given that. A large proportion of comments are neutral, and say all right, well this comment is neutral to me, I'm going to get pretty close to that 80, 90% accuracy with very simple processing. The gold standard is still going to be comparing automated sentiment against manual coding. By manual coding, I mean having Multiple individuals sit down and code statements by hand. Check for inter-rater reliability, just like we would with any survey instrument, and see how an automated program compares to human code. Right, we're seeing rapid improvement in this space. But we need to have that validation. So, human coders are still going to be that gold standard. The reason why we're not going to see human coders deployed at large scale, is just the volume of data that needs to be processed is too high. There's no way that they'd be able to get through all of the comments. It's why we're seeing automated approaches that incorporate human judgement in those decision rules for that, for coding don. When we take a look in the next component of the course, we're going to take a look at a popular tool Crimson Hexagon that has a feature in it, not only do they include an automated sentiment tool, but there's also a component that allows an individual to train an algorithm by classifying a Small fraction of documents and then say I'm happy with my level of classification so far. Let's train the algorithm based on the information that I've provided. And so that's something that's going to become more popular.
