Up to now, apart from having the ability to access the relevant posts, text analytics and Natural Language Processing (NLP) have been the main disciplines required by social media monitoring tools. That is not the case anymore.

Tweets with an image get retweeted 150% more than those without one; they also get liked 89% more. According to Twitter, 77% of all tweets about soft drinks do not have a textual reference to a soft drink brand or anything related to the product category. What?????

So if you are Coca Cola or Pepsi Cola, using social media monitoring tools to crawl the web in order to harvest all the relevant posts about your brands, you will miss out on a great deal of them if you are searching based on keywords only. Even if you just get a hold of the 23% that do include one of your keywords, you will still have no idea what the images included in those are about.

What if the author of a post wrote: “Music and beer…great combination!” and posted the image below?

A social media monitoring tool would never tag this post as one about Heineken. Only a social listening tool specifically developed for marketing insights purposes such as listening247 can offer this capability.

Given all the stats shared above – about the use of images in social media – it is unimaginable how any serious brand owner will continue to only monitor text in social media. Clearly, image analytics needs to become part of the insight management process of any company or organisation using social media listening.

How is it done?

Easy! You need a Data Scientist who can get you a convolutional neural network with over 15 layers – Deep Learning is anything over 4 layers – you find or create a training data set of at least 100,000 images to start with, you get your hands on a VERY powerful computer with multiple Graphic Processors and lots of RAM, or get access to a Big Data infrastructure in the Cloud, and you train a model for a few days. Piece of cake!

What Image Elements can be analysed and why do we analyse them?

There are a number of things that can be analysed in images that can be useful for market research:

1. Logo detection
2. Text extraction
3. Object recognition
4. Facial recognition to detect emotions
5. Theme detection and captioning

Let’s talk about the use of each one of them separately:

1. Logo detection is an obvious one; we need to be able to find the images that include the logos of the brands in the competitive set so that we can extract information and analyse it.
2. Some Twitteratis game the system by using text in an image in addition to the 140 characters that Twitter allows for a Tweet.
3. This is useful when looking for a specific item or product where the logo is not visible
4. Detecting consumer emotions in images where brand logos appear can be very useful in understanding how consumers feel about a brand
5. By knowing what an image is about we can cluster it under its respective discussion driver “bucket”. An image can now be part of a topic taxonomy for holistic semantic analysis.

The obvious conclusion is that text analytics alone does not cut it for social media monitoring anymore; image analytics as described above is necessary, in order to understand what consumers think and feel when they post online. 3^rd Generation Social Listening is here. Click here to experience the DigitalMR “Magic Captioner” and its A.I. magic yourself. 

During the past three weeks I travelled 10 time zones east and west of GMT, to present at three different ESOMAR events:

1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. MENAP Forum 2017 in Dubai on March 22^nd
                                 2. UK member meet-up in London on March 30^th
                                 3. LATAM Forum 2017 in Mexico City on April 7^th

As a souvenir from Mexico City I brought back a broken foot but …hey….no regrets, it was all worth it.

I have been an ESOMAR member for many years, initially as an agency side executive, and now as an entrepreneur, and overall I mainly have positive things to say about the premier organisation of our industry. I have to admit I was a bit worried at the beginning of this decade, mainly about the pace of adoption of innovation, but I think ESOMAR has now fully recovered and is on the ball again.

This is what I spoke about at the three events:

1. The integration of social analytics with retail sales reports and brand survey tracking - together with Nielsen
2. The importance of image processing for theme detection in social listening and analytics
3. Social media listening case studies in LATAM

In all three of my presentations artificial intelligence and machine learning occupied centre stage. The one thing that makes me even more pleased than getting a speaker slot in those events is that DigitalMR was not the only agency that had something to say about the use of AI in discovering customer insights.

The hardest thing when innovation is introduced in an industry is to educate clients to use it effectively. The inertia that we had to endure during the past few years was relentless. Thankfully, the feeling I have after participating in these events is that there is change in the air. The fact that more people talk about AI now, means that we will finally get some traction in adopting these new methods in mainstream market research. After four years of hard work in doing R&D and running pilots with early adopters, we may be nearing the phase whereby the early majority will start kicking in.

Those of us in this field can use all the help we can get to establish machine learning as an acceptable way of analysing big data and integrating it with surveys and behavioural data. Having said that, we have to be really careful as an industry and set some boundaries that will not allow aspiring tech companies to destroy the image and reduce the value of what market research offers as an industry (to its clients).

The same way ESOMAR once created the 28 questions that a client has to ask a vendor before they engage in online research (using access panels) we now need to define the parameters of acceptable market research standards around social listening, the use of natural language processing (NLP), and by extension artificial intelligence. Here is a list of 20 questions that DigitalMR proposes ESOMAR should use as a starting point to create those standards, in a way that is simple and hopefully easy to understand:

1. Are the sentiment classifying algorithms based on Natural Language Processing (NLP) linguistic or statistical methods or both?
2. What is the average sentiment accuracy achievable with the method used?
3. How is sentiment accuracy defined?
4. How exactly is the algorithm trained (if one is used) and how long does it take to get to the maximum achievable accuracy?
5. In what languages can the vendor analyse for sentiment and topics in an automated way?
6. How long does it take to introduce a new language?
7. How is noise (irrelevant posts) due to homonyms removed from the data set to be reported on?
8. Are search terms used or is it an open ended inductive approach?
9. Are posts weighted according to author influence? If yes, how?
10. Is the profiling of people who post (by demographics and other variables) available?
11. How are the harvesting sites selected?
12. How are comments gauged and classified for sarcasm?
13. Is the pricing based on the number of search terms researched?
14. How is the reporting done; what are the deliverables?
15. If Natural Language Processing is used, are adjectives classified as positive or negative in a library (rule based approach to define sentiment)?
16. Is the vendor a technology company or a specialized market research agency?
17. Can specific emotions be detected and analysed? If yes, which ones?
18. Does the vendor use topic taxonomies to identify discussion drivers? What is their semantic accuracy?
19. Is image processing for brand logo and more importantly theme detection available?
20. Can the vendor integrate the harvested data flow with your brand tracking surveys, Nielsen retail reports, or other in-house data sources?

Let us know if you support this initiative and if you have any other questions that you would like to add. As always, feel free to tweet to @DigitalMR and @DigitalMR_CEO.

In 2014, the first time we were interviewed by Forrester about MROCs (Market Research Online Communities), a term they coined, we told them that we were building the first pure DIY MROC platform ever. They were quite impressed and I was too (by their reaction) because I was not aware at the time that none of the known vendors were offering a SaaS that was accessible with zero human interaction. In August 2015, Forrester published “Charting The 2015 Landscape Of Market Research Online Community Offerings” where DigitalMR’s communities247^® featured as one of the 14 tools the report was evaluating.

I find it remarkable that there are only 14 vendors for online communities globally (registering on Forrester’s radar) when at the same time there are over 1,000 social media monitoring tools. I compare and refer to these 2 types of marketing tools specifically because these are the two SaaS products DigitalMR has been focusing on since 2012: listening247^® and communities247^®. Both will be available as DIY SaaS for market research agencies and brands alike.

For some reason, vendors in the online communities space avoid sharing their pricing plans online. I am not sure if it’s because this is a very competitive sector in terms of prices or because there are so many variations of a service like this that it’s difficult to capture pricing in 3-4 set plans.

SurveyMonkey, which is a great example of a DIY online survey tool, was established in 1999. It took the company 10 years to reach revenues of US$ 28 million; now 18 years later its revenue is closer to US$ 200 million with a valuation of US$ 2 Billion. I think SurveyMonkey despite a horrible brand name - in my humble opinion - became a disruptor by opening up the bottom of the pyramid in terms of new users who could not afford to do surveys up to that point in time. At present there are numerous blue chip customers that use SurveyMonkey for their online market research needs, such as Facebook, Salesforce, Samsung, and Virgin America. SurveyMonkey is now in phase II of the disruptive technologies path that Prof. Clayton Christensen has described in his book “The Innovator’s Dilemma”; they started eroding the market share of the incumbent companies. They are now good enough and simple enough for everyone to use.

DigitalMR will take the step that everyone is avoiding. It could be the most stupid move ever, or the most genius one. I guess we will soon find out. Next week for our presence at IIEX Europe 2017 we will introduce a FREE TRIAL Button on our website for communities247^®. There will be no pricing plans to start with, but we will hopefully be able to gauge who is interested in “the SurveyMonkey” of online communities for market research. Another company made this same statement in 2013, but we do not think they have succeeded. Sometimes, timing plays a very important role on the success or failure of a new product. Stay tuned!

Having the ability to know the future turn of events is a human obsession. Unless we are declared prophets by some religion we are not great at predicting the future using our intuition, dreams, and premonitions, or maybe we are but no one is willing to listen. The human mind can handle linear projections well, but exponential…not so much. Humans usually overestimate the short term and underestimate the long-term evolution and progress. Having this thesis in mind, I will attempt to “call” the linear trend interruption of a couple of slow growth technologies by a “hockey-stick” in market research spend.

We are increasingly using advanced analytics and artificial intelligence to aid our intuition and gut feeling about what comes next. We now have ways to consolidate the wisdom of many humans as expressed on social media and other online sites. The wisdom of the masses, the meeting of minds - especially if they are physically close as in churches, football stadia, or live concerts - has a scientifically unexplainable power. If there is a seemingly plausible explanation for this “power” it lies in the spiritual realm for the time being (as opposed to the scientific).

According to Tetlock and Gardner in their book ‘Superforecasting’ (2015) the most successful approach to forecasting is to combine humans (talented?) + a process (an outside view combined with the inside view and unpacking the original question in multiple questions) + computing.

Without a lot of explanation I am listing my 10 new predictions for market research and customer insights below:

1. The total spend on social listening and analytics from market research budgets will be US$ 9 Billion by 2020, up from US$ 2 Billion
2. Social media listening will be about integration with surveys and other data sources instead of a single customer insight source
3. Market research online communities will replace a lot of the “asking questions” part of market research, possibly 50% of all spend by 2020
4. Listen-probe-listen-probe using a social listening platform in conjunction with online communities will become mainstream by 2020
5. Micro surveys that will intercept customers while they perform a relevant action and ask about the experience will grow exponentially by 2020
6. Traditional customer tracking surveys will become a lot shorter in the meantime, until they will at some point during the next 5 years be replaced by a combined approach of intercepts + social listening + online communities
   
7. Artificial intelligence will become mainstream in analysing data for customer insights in the next 5 years
8. A lot of the market research solutions in existence will become available as DIY in the next 5 years
9. As a result of point 8 market research will be democratised as a service i.e. become affordable for SMEs
10. I will chuck this last one in the category of “self-fulfilled prophecies”.  A very powerful notion that has to do more with the persistence and drive of the “prophet” to make something happen. By 2020 DigitalMR will become a global powerhouse in the market research industry or it will be acquired by a global multinational player who will emerge as a winner in the current consolidation wave.

Now you must be asking the question: is this guy a superforecaster (Tetlock & Gardner 2015) according to the definition below?

1. Outside View
2. Inside View
3. Unpack the questions

I will not go through the detailed process on how each of the 10 predictions came to be but I will illustrate it using one example.

Let’s take prediction #1:

A. The Outside View

Outsell, an independent analyst company, predicted the social listening market size to be US$ 5.5 billion by 2020. We know that the human mind is linear and fails to predict the time point of exponential growth. If the growth of social listening was expected to continue to be linear then they would be right.

Innovation in market research historically takes 15 years to become mainstream (examples: CATI, online panels). Social listening started being used by early adopters in 2006. 2020 will be the 15^th year since the beginning; evolution of technologies takes a lot less time nowadays, it becomes exponential and ubiquitous a lot faster than in the past (examples: broadband, smartphones, digital content uploads on the web, mobile advertising etc.). This implies that we could see a “hockey stick” before the 15^th year.

The total market research market including adjacent companies offering technology solutions for market research is US$65 billion.

B. The Inside View

General Mills, Reckitt Benckiser, Heineken, Vodafone, Diageo and a couple of the largest multinational market research agencies globally all asked DigitalMR to demonstrate how social listening integrates with surveys and other data sources. Successful pilots have already taken place.

These companies are not happy with the accuracy of the social media monitoring tools their marketing departments are already using for other purposes.

They are asking for one social listening tool that can handle multiple use cases, including the one for customer insights.

They are all keen to reduce the spend on monthly customer tracking surveys.

Some other younger companies which are more focused on technology are looking for ways to avoid traditional surveys and use big data in a predictive manner.

C. Unpacking the original question: what will be the spend in social listening and analytics by 2020?

1. What is the total market size of market research? US$65 B
   
2. How much of the total MR spend will not be impacted by the rise of social listening? What is left? Retail measurement, Offline Ad effectiveness, Qualitative research (it will become probing), intercepts= US$ 30 B, US$ 35 B left that can be shifted
   
3. What is the current spend on market research and other marketing activities that involve social listening? US$ 2 B
   
4. How many years did it take to get to this spending? 11 years. This is an indication that the trend break to a “hockey stick” is close
   
5. How many companies that currently spend money on market research will invest in social analytics? Conservatively 70%. These will be the largest MR spenders. What % of the relevant spend will be diverted to social media listening by 2020? Conservatively 15%. This adds up to US$ 5.3 B
   
6. How many new entrants will there be in the market research industry from the bottom of the pyramid i.e. SMEs? There are over 10 million SMEs in the US and the UK alone. If 5% of them decided to invest in social listening for customer insights we would be looking at 500,000 companies. How much will they spend per year? DigitalMR is currently a small company and spends about US$ 15K per year on marketing related SaaS. We can assume conservatively that they will spend on average US$ 5,000 per annum. This adds US$ 2.5 B to the estimate of the social listening market for customer insights. For the whole world we can again conservatively add another US$ 1.5 B. The total is US$ 4 B.
   
7. As a result the total market is estimated at US$ 9 B.

As ever we are interested in your views and opinions about these predictions. What do you think? Is the social listening and analytics spend about to take off?