10 Predictions for the Market Research Industry
The next 5 years

10 Predictions for the Market Research IndustryHaving the ability to know the future turn of events is a human obsession. Unless we are declared prophets by some religion we are not great at predicting the future using our intuition, dreams, and premonitions, or maybe we are but no one is willing to listen. The human mind can handle linear projections well, but exponential…not so much. Humans usually overestimate the short term and underestimate the long-term evolution and progress. Having this thesis in mind, I will attempt to “call” the linear trend interruption of a couple of slow growth technologies by a “hockey-stick” in market research spend.

We are increasingly using advanced analytics and artificial intelligence to aid our intuition and gut feeling about what comes next. We now have ways to consolidate the wisdom of many humans as expressed on social media and other online sites. The wisdom of the masses, the meeting of minds - especially if they are physically close as in churches, football stadia, or live concerts - has a scientifically unexplainable power. If there is a seemingly plausible explanation for this “power” it lies in the spiritual realm for the time being (as opposed to the scientific).

According to Tetlock and Gardner in their book ‘Superforecasting’ (2015) the most successful approach to forecasting is to combine humans (talented?) + a process (an outside view combined with the inside view and unpacking the original question in multiple questions) + computing.

Without a lot of explanation I am listing my 10 new predictions for market research and customer insights below:

  1. The total spend on social listening and analytics from market research budgets will be US$ 9 Billion by 2020, up from US$ 2 Billion
  2. Social media listening will be about integration with surveys and other data sources instead of a single customer insight source
  3. Market research online communities will replace a lot of the “asking questions” part of market research, possibly 50% of all spend by 2020
  4. Listen-probe-listen-probe using a social listening platform in conjunction with online communities will become mainstream by 2020
  5. Micro surveys that will intercept customers while they perform a relevant action and ask about the experience will grow exponentially by 2020
  6. Traditional customer tracking surveys will become a lot shorter in the meantime, until they will at some point during the next 5 years be replaced by a combined approach of intercepts + social listening + online communities
  7. Artificial intelligence will become mainstream in analysing data for customer insights in the next 5 years
  8. A lot of the market research solutions in existence will become available as DIY in the next 5 years
  9. As a result of point 8 market research will be democratised as a service i.e. become affordable for SMEs
  10. I will chuck this last one in the category of “self-fulfilled prophecies”.  A very powerful notion that has to do more with the persistence and drive of the “prophet” to make something happen. By 2020 DigitalMR will become a global powerhouse in the market research industry or it will be acquired by a global multinational player who will emerge as a winner in the current consolidation wave.

Now you must be asking the question: is this guy a superforecaster (Tetlock & Gardner 2015) according to the definition below?

  1. Outside View
  2. Inside View
  3. Unpack the questions

I will not go through the detailed process on how each of the 10 predictions came to be but I will illustrate it using one example.

Let’s take prediction #1:

A. The Outside View

Outsell, an independent analyst company, predicted the social listening market size to be US$ 5.5 billion by 2020. We know that the human mind is linear and fails to predict the time point of exponential growth. If the growth of social listening was expected to continue to be linear then they would be right.

Innovation in market research historically takes 15 years to become mainstream (examples: CATI, online panels). Social listening started being used by early adopters in 2006. 2020 will be the 15th year since the beginning; evolution of technologies takes a lot less time nowadays, it becomes exponential and ubiquitous a lot faster than in the past (examples: broadband, smartphones, digital content uploads on the web, mobile advertising etc.). This implies that we could see a “hockey stick” before the 15th year.

The total market research market including adjacent companies offering technology solutions for market research is US$65 billion.

B. The Inside View

General Mills, Reckitt Benckiser, Heineken, Vodafone, Diageo and a couple of the largest multinational market research agencies globally all asked DigitalMR to demonstrate how social listening integrates with surveys and other data sources. Successful pilots have already taken place.

These companies are not happy with the accuracy of the social media monitoring tools their marketing departments are already using for other purposes.

They are asking for one social listening tool that can handle multiple use cases, including the one for customer insights.

They are all keen to reduce the spend on monthly customer tracking surveys.

Some other younger companies which are more focused on technology are looking for ways to avoid traditional surveys and use big data in a predictive manner.

C. Unpacking the original question: what will be the spend in social listening and analytics by 2020?

  1. What is the total market size of market research? US$65 B
  2. How much of the total MR spend will not be impacted by the rise of social listening? What is left? Retail measurement, Offline Ad effectiveness, Qualitative research (it will become probing), intercepts= US$ 30 B, US$ 35 B left that can be shifted
  3. What is the current spend on market research and other marketing activities that involve social listening? US$ 2 B
  4. How many years did it take to get to this spending? 11 years. This is an indication that the trend break to a “hockey stick” is close
  5. How many companies that currently spend money on market research will invest in social analytics? Conservatively 70%. These will be the largest MR spenders. What % of the relevant spend will be diverted to social media listening by 2020? Conservatively 15%. This adds up to US$ 5.3 B
  6. How many new entrants will there be in the market research industry from the bottom of the pyramid i.e. SMEs? There are over 10 million SMEs in the US and the UK alone. If 5% of them decided to invest in social listening for customer insights we would be looking at 500,000 companies. How much will they spend per year? DigitalMR is currently a small company and spends about US$ 15K per year on marketing related SaaS. We can assume conservatively that they will spend on average US$ 5,000 per annum. This adds US$ 2.5 B to the estimate of the social listening market for customer insights. For the whole world we can again conservatively add another US$ 1.5 B. The total is US$ 4 B.
  7. As a result the total market is estimated at US$ 9 B.

As ever we are interested in your views and opinions about these predictions. What do you think? Is the social listening and analytics spend about to take off?


A blog post written at 300 km/h

A blog post written at 300 km/h

As promised last week, here I am writing another blog post as I travel at 300 km/hour on the Eurostar towards Brussels - from London - on a Sunday afternoon. I am heading to my second consecutive participation as a speaker at LT-Accelerate; a conference about language technologies, not the usual market research conferences that I attend.

Last year at LT-Accelerate I spoke about rich analytics for social listening and stressed the importance of semantic analysis and accuracy; this year I will be describing what differentiates the more than 1,000 social media monitoring tools currently available out there.

Looking at this from a market research and customer insight perspective, we categorised the social listening tools into three generations:

  1. GEN 1: sentiment accuracy less than 60%, search based topic analysis, limited attention to noise elimination, automated sentiment analysis in usually one or two languages only
  2. GEN 2: sentiment and semantic accuracy over 75% in any language, inductive approach to report topics of conversation, significantly reduced noise (less than 5% irrelevant posts)
  3. GEN 3: In addition to what Gen 2 social listening tools can do, those few that can be classified as GEN 3 can also detect emotions, analyse images in an automated way for brands in terms of theme and possibly sentiment, and they offer guidance for integration with consumer tracking surveys and other data sources and profile users.

If you want to know what generation your current social media monitoring tool belongs to, all you need to do is ask your vendor what is their sentiment and semantic accuracy and whether they can detect emotions and analyse images for insights.

The main reason I go to conferences such as this one is to demonstrate thought leadership in the field of market research and customer insights, with the hope that prospective clients, partners, and vendors will come forward and initiate conversations that could develop to become mutually beneficial deals.

Last year only half of the conference delegates showed up because of the terrorist attack that had happened in Paris. Brussels was on a high terrorist alert that started the Sunday before the conference; the prudent thing to do was to stay at home and switch to a skype presentation as some speakers did. My take on the situation was that a city is at its safest when it is on high alert, so I decided not to change my plans. Indeed as I arrived at the train station last year and on the way to my hotel the streets were deserted, apart from armed soldiers. It was eerie but funnily enough it felt quite safe.

So here I am again this year on my way to the Brussels Central station and in the absence of a red alert due to terrorist threats I sort of feel less safe. I am making a mental note to remain vigilant and pay attention to what is going on around me; look out for any suspicious behaviour in other words.

Enough reminiscence, back to the essence of this post: I am sure there are other meaningful ways to categorise social listening tools and I would be very interested to find out how other people classify them. Maybe a plausible way to classify them is according to the use case of each one. Maybe another is the target customer/department the tool was created for, such as:

  • PR
  • Communications
  • Operations
  • Customer Service
  • New Product Development
  • Customer Insights

In my opinion around 98% of the current tools on the market belong to Gen 1, around 1% belong to Gen 2, and only a handful belong to Gen 3. I would not be the least surprised if the only social listening tool that meets all Gen 3 criteria is listening247®. Clearly, only Gen 2 and 3 tools are suitable and can be used for market research and customer insights. Gen 1 tools would be disqualified from the get-go, if nothing else, due to the noise (irrelevant posts) that is analysed and included in what is reported to the user as relevant.

How do you classify social listening tools? Please feel free to share your approach with me on Twitter @DigitalMR_CEO.

A blog post about…writing a blog post!

I know I should be writing at least 4 blog posts per month, but somehow I only manage to do one. Call it procrastination, call it daily re-prioritisation of tasks, whatever the reason it does not matter. One of my favourite business truisms is: “There are reasons and results, reasons simply don’t count”.

Especially if you think that there is a number of reasons (no pun intended) as to why it is beneficial to write frequently on your company’s website or blog:

1)      You hopefully demonstrate thought leadership in your subject of expertise

2)      You improve the SEO for your website

3)      You can initiate a dialogue with prospective customers or other stakeholders

Reasons 4 and 5 are more personal and can be seen as the icing on the cake: it can be fulfilling and if you aspire to publish a book one day you may discover that you have already written it bit by bit without making a big deal out of it.

My subject of expertise is Online Market Research with special focus on “social listening” and “online communities”. The former is searched for on Google 480 times monthly in the UK only and the latter 390 times – this is what HubSpot says by the way. Both combinations of words are currently on the second page of Google search for DigitalMR. One of the reasons I am mentioning them in this and other blog posts is SEO; I would really love it if we moved up a few places in the Google ranking for these keywords. It could increase the monthly visitors of the DigitalMR website by at least 1,000 if you think that people also use Google in the US and many other countries, to find information on social listening and online communities in the English language.

There is also the issue of the personal brand which apparently is very important for leaders of start-ups. A good friend and advisor told me recently that tech start-ups are the new rock bands. When I was growing up we would look at photos on album covers of rock bands and get inspired, now we look at photos of Zuckerberg and Musk to get our cues on what is trendy.

I was actually planning to write a blog post about the US elections this weekend; instead you are getting a boring introspection. We did a poll on Twitter on October 3rd and the outcome was that Trump received 53% of the votes Vs 47% for Clinton. I did not like the result and I also did not believe it to be honest so we said nothing about it. Now I felt compelled to say: our Twitter poll predicted the US elections result…but I am not going to do it; you see how I did this Smile?

The point is, there is saturation in the media about the subject…. Duh….no one is interested about another Trump story. So now we are getting somewhere; the reason why I am doing an introspection piece is because I want us to be unique and different and offer original ideas that are not recycled a million times.

How am I doing? If you are still reading and you are not my mom or my wife (I don’t think she reads them either) then maybe I am on to something. I will be checking on Google Analytics and HubSpot all of next week to learn something about this style of writing a blog post. I do like A/B marketing tests; I do not know how I will turn this one into a test but I need to figure something out during the next 10-20 lines so that I can offer some take-home value to you the reader.

I am guessing I will write another blog post next weekend on the Eurostar on my way to Brussels for the LT-Accelerate conference. That one will be more about social listening or online communities – see how I did this again? Well don’t get too excited, it is not that easy; it is not enough to mention a keyword several times in a page in order to rank on Google. As a matter of fact, I might be penalised by the Google algorithm if the mention of any keyword is more than 3% of the whole document – so I heard, but you can never be sure with the Google algorithms. The latest is Penguin 4.0 and before that we had Possum and Panda. I may need another few sentences so that the mention of social listening will become 2.9% of the whole post. The previous word was number 749; 3% of that is 21 times; I think we are safe I only mentioned social listening and online communities 4 times Smile.

As always, if you have any comments feel free to share them with me on Twitter @DigitalMR_CEO.

Does the Market Research Industry Need a New Name?

Market Research

When I read titles such as this one in articles or blog posts, I expect that the author’s answer is a YES, and I prepare myself to find “holes” in their arguments. So I invite you dear readers to wear the hat of the sceptic, and share your views about this existential question (in my opinion anyway).

I have always had a problem with the term “market research” because it is different but close to “marketing research” and it is not an all-inclusive term for what ESOMAR or MRS or CASRO consider our industry to be. Over the past few years and knowing what I know about the type of research made available through the use of technology (such as social listening and online communities) my concerns about this name being unrepresentative have only grown.

If the job titles of the people who carry out MR within Brand organisations are an indication then Consumer Insights or Customer Insights should qualify as new name contenders. “Market Research” is not (just) about the number of responses, it’s about the insights, so shouldn’t the name of the industry be more representative in this sense?

The limitation of market or marketing research as names is that they only go half way. What does that mean? Take a look at the following data operations that are involved in the full spectrum of market research activity:

  • Collection tool set-up
  • Collection
  • Cleaning/validation
  • Processing
  • Analysis/Synthesis
  • Action Discovery
  • Visualisation

Market Research sounds like it is over when the data is collected and maybe understood, but what about the rest of the activity that takes place in order to discover the elusive insights? We spoke before about ‘insights’ being a buzz word with different people understanding different things. Our definition however is very clear: an insight is a “nugget” that can usually be discovered by synthesizing information from more than one source and by adding a good measure of intuition; it is actionable and it delivers a positive result when actioned. It is not a number that can be extracted from a single market research report.

The need to rename our industry becomes more acute nowadays when a lot of data (Big Data) is easily accessible by organisations and thus commoditised. The use of Artificial Intelligence and specifically Machine Learning makes it so much easier to look for insights in big data-sets. We can get data by asking questions (traditional market research e.g. surveys), by “listening” on social media, by tracking behaviour through transactions, or by observation. The best approach is actually to integrate all of the above sources in order to increase our chances of discovering unique business insights.

It looks like the word “insight” is mentioned a lot in this post... Perhaps it’s a giveaway on the new name for the MR industry? Well ‘insight’ is good but ‘foresight’ is probably even better. Does the current market research industry have what it takes to be a player in foresight generation? Instead of using rear view mirror techniques such as what Business Intelligence prescribes, can our industry make the leap toward Predictive Analytics?

Maybe we as an industry are a bit late in the game; the IBMs and the TeraDatas of this world are already players in this space and have been for a few years now. In any case the first step toward any change is introspection and soul searching. What is the role of our industry? If it still is to gather data then we should continue calling ourselves market researchers but if it is going to be more than that, then here are some candidates to trigger debate:

  • Insight Management
  • Foresight Research
  • Insight to Foresight Management
  • Business Decision Support
  • Organisation Decision Support

Changing the name will be the first step towards accepting some of the real changes that are taking place; the new truths that define who we are.  Please feel free to tweet me on @DigitalMR_CEO if you have any other ideas to suggest on the re-naming of the MR industry, or to let me know if you disagree. I look forward to learning more about your views and opinions on the above.

What is Market Research and What isn't? What is Ethical and What isn't?


These are some really big questions, probably too big to handle in one blog post. We have to start somewhere though; this is too important to let it pass without discussions that should ideally involve as many market research practitioners as possible – both on the agency and the client side. The real trigger to write this blog post, however, is the 7th R&D grant DigitalMR has won, this time as part of a consortium of 7 entities: two from Portugal, two from Spain, two from the UK (DigitalMR and City University) and one from Germany. The project is called DiSIEM (Diversity enhancements for Security Information and Event Management) and it is funded by the Horizon 2020 framework of the European Union. This is a trigger because a whole work package has to be delivered at the front of the project about ethics; a very important element of humanity. The handling of these questions should permeate all industries and market research is not an exception.

With social listening & analytics now in the picture, along with behaviour tracking, the lines of where market research stops and marketing starts are getting a bit blurry for me. I hope it is just me. If you have a clear view, then please share it with the global insights community because some other people and organisations seem to be confused too.

It looks like the market research associations are trying their best to get ahead of this new world with all the new disciplines and its blurry lines. ESOMAR seems to be getting its cue from the EU and sometimes it tries to influence its legislation. I am not sure how successful they are.

Here is an excerpt from the ESOMAR guideline for social media research:

“Researchers must not allow personal data they collect in a market research project to be used for any other purpose than market, social and opinion research. If it is intended to collect personalised social media data for other purposes, they must clearly differentiate this activity from their research activities and not misrepresent it as research.”

This is not very different from what is in the code of conduct for market research, so no surprises here. My interpretation is that a company in this space IS ALLOWED to collect personalised social media data for purposes other than research - as long as they clearly differentiate it from market research and not misrepresent it as such.

Now that the rules around collecting data are clear we can concentrate on reporting personally identifiable data. Here is another excerpt from the ESOMAR Guideline:

“Social media platforms offer many opportunities to view personally identifiable data. Some people post information that overtly discloses their identity, are aware of this and have a diminished expectation of privacy. Others are not aware that the services they are using are open for others to collect data from or think that they have disguised their identity by using a pseudonym or username. However, online services are now available that make it possible in many cases to identify a “poster” from their username or comments and can link that to many other aspects of personally identifiable data including their address, phone number, likely income and socio demographic data.”

Well I am not sure it is a market research practitioner’s problem if someone is not aware that she/he is posting on a public website even though it is common knowledge (e.g. on Twitter) and openly explained on the relevant websites and their Terms of Use (ToU).  

…and I am sure I totally disagree with the next excerpt:

“Given this, data cannot always be 100% anonymised on the internet by merely removing the username and linked URL from the comment. Therefore if researchers wish to quote publicly made comments in reports or to pass these on to people not bound by the ICC/ESOMAR Code (or a contract linked to this), they must first check if the user’s identity can be easily discoverable using online search services. If it can, they must make reasonable efforts to either seek permission from the user to quote them or mask the comment to such an extent that the identity of the user cannot be obtained.”

I am not quite sure why the market research associations feel this way. I assume it is because they want to extend the survey thinking into anything else that is market research nowadays. However there is a big difference between surveys and social listening. In the survey world market research agencies actually collect the personal information thus they are responsible to safeguard it based on the codes of conduct and the law. In social media research, agencies have nothing to do with putting the personal information in the public domain, so why should they be responsible to mask it or anonymise the posts?

I asked this question many times before but I have not received a satisfactory answer yet: What about the copyright laws? Doesn’t every author of any published content have the right and expectation to be quoted when the content is quoted?

Of course in the guidelines we also found this sentence:

“If consent has not been obtained (directly or under the ToU) researchers must ensure that they report only depersonalised data from social media sources.”

This effectively allows a market research agency to report personalised data from e.g. Twitter if Twitter’s ToU state that by publishing on their medium users should expect to be quoted with whatever personal information they have shared. Hmmm confusing….which one is it: de-personalise or give credit?

Tweet to @DigitalMR_CEO if you have an opinion!