The Impact of Ambiguity in Social Listening and Analytics

SarcasmThere are many forms of ambiguity in social media posts, with the most popular being sarcasm. Sometimes it is confused or used in an interchangeable way with irony. Here is a definition for the two terms from :

Irony is used to convey, usually, the opposite meaning of the actual things you say, but its purpose is not intended to hurt the other person. Sarcasm, while still keeping the "characteristic" that you mean the opposite of what you say, unlike irony it is used to hurt the other person.”

For the purposes of this blog post, both irony and sarcasm are responsible for and present the same problem when trying to automatically annotate a post with sentiment or an emotion. The author of a social media post may write something positive about a brand e.g. “I love the new flavour” but if it’s sarcastic, then it is really a negative post, and vice versa e.g. “don’t you hate this ice cream flavour?”.

DigitalMR’s claim to fame, since 2014, is that its R&D focus to solve the problem of low accuracy in automated sentiment analysis in any language, has produced a solution – listening247 – that delivers over 80% sentiment and semantic precision (precision is one of the accuracy metrics in big data analytics). The reason why it is not and cannot really be 100% is because of ambiguity. The outcome of ambiguity in this context is that humans will not agree amongst themselves about the sentiment of a sarcastic or ironic post. Some will think it is positive, some may think it is negative, and in some cases others will think it is neutral for the brand mentioned in the post – i.e. the sentiment is not towards the brand but something else (see Fig.1); it follows that we cannot expect an algorithm to produce a result that everyone agrees with in a case like this. In our research, we have found that on average 10%-30% of posts about a category contain some form of ambiguity. In the example below, 43% was the highest level of agreement among 30 market research practitioners; this is why 80% precision is an excellent result for automated sentiment analysis.

Manual Sentiment Curation of an Ambiguous Tweet                    Figure 1: Manual Sentiment Curation of an Ambiguous Tweet (Base n=30)

Some of you are already aware that DigitalMR uses machine learning to annotate sentiment in an automated way. Machine learning implies that there is an algorithm or a combination of multiple algorithms which are trained with the use of a training dataset, to create a model that does the job. There is one possibility to expect 100% sentiment precision; If supervised machine learning is used (as opposed to semi-supervised or unsupervised), it means that humans create the training dataset manually. If only one human is responsible for creating a training dataset, then the model will only use this person’s judgement to annotate posts for sentiment. In a case as such, because only one person has to agree with the sentiment annotated by the model, if that person is the judge of the model’s precision – then 100% precision is achievable – because that person will not disagree with herself.

It is needless to say that when machine learning is used for automated sentiment analysis, by definition, the identification of sarcasm/irony is a solved problem. “Why?” you may ask. Because a human curator (the person who creates the training dataset) has an understanding of sarcasm/irony, and more often than not, he or she will detect it and annotate a post accordingly.

Will it take Social listening and Analytics 15 years to become mainstream in Market Research?

DigitalMR CEO Michalis Michael at IIeX NA

Some quotes - especially the ones predicting the future, can become really famous until they expire or they are proven wrong. Do you remember the March 2011 quote from Joan Lewis – Global Insights Head of P&G at the time? It was about the future of market research and was shared at the ARF conference. Apparently she said: “Survey research will decline dramatically in importance by 2020, with social media listening replacing much of it and adding new dimensions”. I am not sure why but I personally took this quote to mean that more than 50% of survey research would be replaced by social media listening. I must have used this quote hundreds of times in client presentations and conferences since then. 

We are now halfway through 2016 and we have 3.5 more years for this quote to become true. According to a February 2016 report published by Markets and Markets, the global social media analytics market was US$ 1.60 B in 2015. The report predicts that the market size, which includes both SaaS and professional services revenue, will grow to US$ 5.40 B by 2020. This represented 2% of the total market research market in 2015; it may be over 5% by 2020. These percentages are nowhere near the 50% mark that I took Joan’s quote to mean.

What is really interesting is that after so many years of presenting the quote, I recently met Joan Lewis in person for the first time at the IIeX 2016 conference in Atlanta. I rushed up to her after her presentation to take my turn in talking to her. My understanding is that she is now a freelance consultant. I told her that I have used her quote multiple times and that it was an honour to finally meet her. My question to her was – yes, you guessed it right – “do you think your quote will come true in 3.5 years?”. I explained that since 2011, when the social listening and analytics market was probably around US$ 100 million, it has grown 16 times its size, which is a steep growth from a low base, but not an exponential growth as expected. “What happened?”, I asked with genuine curiosity; her response was that by ‘listening’ she meant all ways of listening to the customer and not just social. I must say I was a bit disappointed with this response.

Social Listening and Anlytics market size

Here is my take on what happened and is still happening: Customer Insights Managers within organisations were slower to adopt the new methodology of data collection and analysis because the tools they tried out were not accurate enough. The sentiment precision was below 60% at best and there was no semantic analysis the way they needed it to be for integration with surveys. This led to low trust on anything that was called ‘social listening’, even if it was offered by a legitimate next-gen market research company. When companies like DigitalMR came out with the result of a sentiment and semantic precision of over 80% in any language after multi-year focussed R&D work, it was almost unbelievable for end-clients. The reason being, that they had already tried 3-4 social media monitoring tools by that time and none had delivered on expectations. Also, there are not enough of us to spread the word that “YES IT IS POSSIBLE TO ACHIEVE ACCURACY LEVELS THAT MAKE THIS TYPE OF ANALYTICS ACCEPTABLE FOR CUSTOMER INSIGHT PURPOSES”

I now have a new favourite client quote, this time the quote was told to me personally and I had the chance to discuss and understand the intent and purpose of the quote. Tom Emmers, Senior Director Global CMI of Heineken, said: “I believe that today, given the rapid advancement of capabilities, we should only resort to surveys if we cannot find the answer in social listening or behavioural tracking data”. He understands that this change will not happen from one day to the next but he, just like us, believes that this is the direction that the customer insights function is taking. We all understand the respondent sample challenges of our times with diminishing response rates, the system 1 and system 2 considerations expressed by Daniel Kahneman (in his book Thinking fast and thinking slow) and the bad quality of consumer access panels. Open-minded and innovative customer insights practitioners are now looking to integrate social analytics with brand tracking surveys and behavioural tracking in order to enhance their insight generation process. Then, at some time in the future, when social media will be more ubiquitous, unsolicited customer feedback and behaviour may even replace surveys completely. For what it is worth, I believe that the total market size for social listening and analytics will be over US$ 7 B by 2020 (see Fig. 1). Feel free to ask me (@DigitalMR_CEO) why I believe this to be the case!

What is the Use of Social Listening in Business

The Use of Social Listening in BusinessUnlike other data collection methods for market research, social listening is not just about customer insights. Uncovering unique actionable insights is the reason why social listening is relevant to market research, however, there are many more use cases for various departments within an organisation.  

Here is an outline of 8 use cases with the first one (customer insights) also being applied across all the other 7:

  1. Customer Insights
    Asking questions in surveys and focus groups is no longer enough in order to uncover customer needs and wants so that organisations can stay relevant. Unsolicited posts on the web provide a different flavour of information, the value of which becomes exponential when integrated with tracking surveys and behaviour. 

  2. Advertising
    Positive customer testimonials on social media can be used both to strengthen advertising messages and also as “the reason to believe” when Unique Selling Proposition (USP) claims are made in Ads.

  3. Public Relations
    By discovering discussion drivers on social media, organisations can publish relevant content, increasing the probability that it will be propagated by customers themselves. They can control the narratives and they can create new ones that serve their performance goals.

  4. Customer Service
    Nowadays it is an expectation that customers are able to tweet a need or a complaint and receive a response from the organisation they are addressing within minutes. Public relations and customer service on social media are somehow interlinked. It is as important to appear to be tackling customer issues head on and transparently, as it is to actually fix them!

  5. Operations
    Fixing product or service issues communicated by customers on social media is critical from an operations and PR perspective. Multi-branch organisations in particular, can benefit from feedback on social media about individual branches. This use case may render mystery shopping obsolete.
  6. New Product development
    Customers tend to post about product features that they are not happy with or that they are missing. They also post about their pains, key information that can lead to innovation in order to address these pains.
  7. Board of Directors
    One of the things that are of interest to the board is Corporate Reputation,which can also be part of Public Relations. Social listening provides a unique opportunity to board level executives to keep their finger on the pulse of their customers in the most direct and efficient way.
  8. Risk Management
    A problem with a product or a service can easily be blown out of proportion on social media. Companies need to keep their ear to the ground and identify issues before they become real crises that will impact brand equity in a negative way.

In order for any and all the above use cases to be valid and useful for an organisation, it is an absolute pre-condition that the sentiment and semantic (topics) precision of the social listening solution used, is the highest it can be. As an example, if a social media monitoring tool feeds negative sentiment posts about a brand to its customer service team with only 50% precision, this means that the team will have to sift through all the posts to actually find the ones they should really respond to. This can be frustrating for the team, and it also means that the organisation will have to spend more time and resources in order to deal with customer service on Twitter or other social media platforms.

Over 80% sentiment precision and 85% topic precision is what is achievable – as already mentioned in previous articles in more detail – thus organisations do not and should not have to settle for less these days. Please do share your experiences with social listening tools if you have used them, either for the above use cases or others that are not mentioned here.


Walking the Talk

The Positive Effect of NegativitySince 2011, some of us have been telling everyone who is willing to listen, how important social listening and social analytics is for their business. It would of course be disingenuous not to do it for ourselves, so we did…

662,575 English posts were harvested about Market Research from Twitter, Blogs, Boards (Fora), Videos, and News, posted anywhere in the world between July 1st and December 31st 2015. 29 company names were included in the harvest query, along with some key terms....

Article no.5 of the Social Media Listening series by @DigitalMR_CEO is out on ESOMAR Research World Connect. Stay tuned for more.

A new marketing phenomenon: The Positive Effect of Negativity

The Positive Effect of Negativity"When we came across the Coca-Cola Superbowl ad paradox, we investigated and analysed it to death. We then started thinking about similar cases that may support our hypothesis i.e negativity brought out by a relative minority, under certain circumstances, may have a positive effect through awakening the majority.

What we discovered had actually happened was almost unbelievable, at least for it to have occurred organically..."

Article no.4 of the Social Media Listening series by @DigitalMR_CEO is out on ESOMAR Research World Connect. Stay tuned for more.