Caveat Emptor, Buying Research in the Era of Just In Time Publishing

Caveat Emptor, Buying Research in the Era of Just In Time Publishing

The practice of “Just In Time” report publishing has emerged in the market research reports business over the past few years. The providers will advertise reports with titles loaded with keywords for SEO purposes and descriptions created via AI tools.  The problem is that the reports do not actually exist.

For market research buyers the product they may order is rapidly assembled from previously issued reports and updated with content sourced via web scrapers and bots. It would be one thing if providers had significant amounts of quality content and forecasting that could be drawn from to issue a customized inquiry. It is quite another when the provider’s content is of lower quality and lesser value.  The key thing to consider is the provider.  A large-scale firm with significant market presence, prestige and a track record can provide much greater value and less risk than the research factories operating offshore.

So how can you tell if the product is real or waiting to be assembled? 

A sample should be readily available, with content more substantive than boilerplate, no blanked-out pages and something that shows what might be available.  If the provider cannot show an actual sample, it likely means the report needs to be assembled.

If there are several reports available within their coverage sector(s) (often long tail) with several keywords and phrases in the titles, the provider is likely engaging in factory practices.

Tables of contents and descriptions look similar, with minor variations within sections.

The provider claims to have several hundred reports available that are all “recently updated”  Impossible given the economics and market conditions.  Sectors like PC, semiconductor or consumer electronics can be updated quarterly but that product is more obvious to the buyer.

The offer of X% customized for prospective buyers?  Factory assembly job

No name analysts listed on the reports or on the site.

If there are analysts associated with the product, how many have they produced within a certain time?  Analysts can reasonably issue 700 pages of content per year in addition to the normal research functions they engage in.  A high number of reports with 150 pages is a red flag.

Buyer Advice

While this is not an exhaustive list, it should present buyers with some things to consider.  Corporate buyers in charge of sourcing data and content owe their internal stakeholders the best decision support materials available.  Individual buyers may not be aware of what the industry practices are and should take note of the items listed above.

Stakeholders should insist on a bio of the analyst(s), talk to them for 15 minutes, ask some hard questions on availability of the product, ask them to explain how they can produce what they claim to have available in such timeframes.  Talk to the analyst to see if they have any expertise in the space as well.  Market research is fine but if they aren’t subject experts, why bother?

Market reports used to be a great way for companies to access forecasts, competitive intelligence, market feedback and analyst takes.  As market research factories have grown in scale and market presence, the value of reports has declined due to the practices employed within these shops.  Buyers are left to sift through piles of poor-quality products and do not necessarily understand the hidden risks.

 

A little awareness is a good thing….

The New Tech Shuffle, Are They an Analyst or a Paid Optimist?

The New Tech Shuffle, Are They an Analyst or a Paid Optimist?

New technology is where the action is. A lot of excitement, money, endless potential for innovation, disruption, and money. The perfect place for the Optimist to do their trade.

The Optimist gets paid for the sunshine they spread, not the forecasts of risk, negativity about companies’ downsides or alternatives that don’t support what they are advocating for. The analyst isn’t attached to the future outcomes in the way the Optimist is. The analyst is more concerned with their professional reputation.

There isn’t anything wrong with analysts being optimistic about market opportunities presented by new technology introductions or advancements. If there is a need, a means, and a viable business case then by all means, be bullish on the future.

But, optimists also never seem to the learn the lessons that the Gartner Hype Cycle teaches (https://www.linkedin.com/pulse/8-lessons-from-20-years-hype-cycles-michael-mullany/) They are reliable for getting on or in front of the next big thing, whatever that is and having a track record of issuing hockey stick/exponential forecasts and market outlooks that that ignore factors or historical frames of reference.

Red Flags:

Market reports or forecasts where everything grows and with a decided major ramp in 3 years and the market quintuples in value by year? If this happens over time you are dealing with an optimist. Hard pass if you are smart.

No negative comments or controversial takes or tries not offending anyone that might buy their stuff or they have bought in hard and won’t point out the risks?

When the quarter is down or there is a pattern of decline, do they step back from previous comments and offer new guidance or do they double down on the rebound that is coming? Do their comments resemble word salad to hide the bad news?

They produce vendor events that sell the dream vs the more objective reality of the market. The more optimism and future riches they profess the less you should trust them.
If they sell vendor advertising on their analyst platform, that is a clear sign they are not really in the analyst business. If the firm is aligned with a media outlet, be cautious as well, given the business considerations present. In addition to advertising, media platforms sell sponsored content. Pay to play isn’t good for independence.

On the other hand, if they offer paywall commentary for the good stuff then take that as a sign that there may be some objectivity possible. A buyer should be able to sort out what is there though.

If they offer advisory services, then the person or firm is most likely going to be an analyst as they must be able to push back on bad ideas or strategies or steer a client in a more beneficial direction. The optimist is far too risky to work with as an advisor as they only see the upside for vested interests or their having bought in on the vendor and industry press hype.

When you engage the Optimist, they tend to stumble, deflect or even gaslight to some extent when presented with contrarian views or are forced to explain their own when challenged. They will not own their hype tendencies.

The analyst, on the other hand, has a POV they can support and will adopt new thinking when presented with new evidence. They will continue to dig and look for ways to rebut their own analysis to make it more sound and provide value to clients and the broader market.

Will AI End the Market Research Factories Once and for All?

Will AI End the Market Research Factories Once and for All?

The arrival of LLMs such as ChatGPT, Bard and the ever-growing universe of AI tools that emerge daily, has stirred conversations as to their impact on industry sectors, business processes, people and more.  This article discusses how the widespread usage of the technologies will impact the market research and report space.

 

AI dominates the business and social media content news landscape.  Every day we see stories of new SaaS apps becoming available, advances in ChatGPT prompts, Google, Meta. Amazon, Microsoft, and others engaging in an arms race, and how Generative AI is being integrated into anything and everything.  Hard to keep up, isn’t it?

 

The upsides are there. Productivity gains, more output, more automation, and new use cases that are pushing the limits and driving innovation.  Tools that can search, summarize, rewrite, “analyze”, spot trends and even create reports for users. If the user understands the limitations of the tools and how they search, source and access data the risks are minimal.

 

One of the major keys to the ensuring best outcomes is via data observability (https://www.decube.io/post/data-observability-llm-accuracy) which is critical within within the analysis realm as the models only work when one can verify data sources, collection methods and accuracy.  Layering AI on top of a defined data reservoir makes a lot of sense.  Utilizing AI chatbots to support research and analysis is another story though.

 

Market research factories that make their money assembling reports using data scrapers or who engage in copy pasta are now turning to AI tools to support their “Just In Time” production model of posting reports on websites (their own or via aggregators) and backfilling the orders.  AI tools are more of a means to spitting out end products that are low quality, lacking in context and worse, inaccurate given the issue of LLM hallucinations where the ChatGPT, Bard etc. provide inaccurate results.

 

The Just In Time model is highly problematic as the factories will compile 200 plus page documents in under 3 days packaging together previously created forecasts based on data that could very well be AI impacted along with data and content collected, synthesized and summarized utilizing these tools.  And with the goal of having a report available for just about any subject or segment it is impossible for report factories to avoid these practices.

 

It also isn’t possible for a firm to produce 800 reports a year despite what they list on their website.  The economics simply don’t prove out.  A production output at that level normally requires 200 people in research, forecasting and analysis, especially if the firm sells 200-page reports.  Add in the sales, marketing, production, management and overhead and even with offshore labor benefits the company must generate nearly $8 million a year in revenue to break even.

 

AI tools are a way for them to shave costs while giving the illusion of extensive product offering and market coverage via a team of analysts that are not obvious to anyone.

 

Many companies buy reports for 3rd party validation of market sizing, investment justifications, internal funding, M&A, or market perspectives. Given the issues with AI and the low standards employed by market research factories, how can a buyer or internal user feel confident in the content or numbers produced by factories or even know what they are getting though?

 

The answer…

Work with established research brands where the buyer is able understand the vendor’s database, research methodology and the analysts writing the report.  Quality firms can produce the analysts and present their processes in an obvious way, factories cannot.

 

Buy directly from reputable brands and avoid the factories and the resellers who carry these products.  While I am aware that some companies do screen the publishers, if the products sell then they stay in their catalogues.

 

If the report is not available when the buyer inquires, don’t wait 3-5 days for a scrape and compilation job. That should tell the buyer the product is a factory job.

 

Master the AI tools internally and not only make sure that the data and content sources are up to user standard but do your part to avoid supporting firms that lessen the value that good firms provide.

So Now What? Navigating Through a Pandemic World

So Now What? Navigating Through a Pandemic World

The global economy is engulfed in the aftermath of an epic and truly horrible black swan event that is shutting down entire countries and economic activity across multiple industries and countries. I wrote something a while back about recessions and what firms need to consider in navigating them. My suspicion was that an economic slowdown was inevitable, and it was time to start planning for the pain that would result. This is completely different.

While some of the post’s premises hold, perhaps time to look at how to respond to the current crisis with a different set of urgency. Getting a full measure of what you are facing and how your firm or practice will respond will make the difference between whether you survive. This is not hysteria or hype, I have been through the 2000-2003 tech market crash, the 2008 credit crisis and a number of market and industry specific dislocations that washed out a number of firms. Firms with talented people and decent to superior reputations…

My advice? Before you can get to an action plan there are some stages a firm or personal practitioner needs to get through.

Step One: Wrap your arms around the notion that the economy has suffered a massive body shot and has little support under it and is falling fast. Airlines, railways, automotive, banking, travel, and hospitality, you name it, things are in the toilet. Medical is in high demand but…. communications infrastructure and services, gaming and digital entertainment and logistics are not in free fall either. Unemployment numbers are going scale like you have never seen and despite the efforts of the central banks and government to provide support the longer the shutdown continues the more pain we will collectively experience.

Step Two: Whatever you have said or published up until this started? Yeah, it is mostly useless now. I know that is a hard pill to swallow but the faster you accept that then the more easily you can move forward. Time is the asset for professional service people no matter what the angle. If you have spent six weeks researching and writing a report with forecasts, consider them as something you will need write off.

Step Three: Accept that your clients and markets are in a panic situation and all deals, sales pipelines or projects are pretty much dead on the vine. Consider that you may have to reduce prices, package things or rework what you have published.

Step Four: Know that your revenues are likely going to crater in short order and your business is in trouble either now or down the road depending on how you are structured (annual subscription vs quarterly vs ad hoc vs consulting engagement driven). Firms that are flush with annual renewals that start on January 1st are going to fare better than others. That does not mean renewals are going to be easy next year. They will not.

Step Five: Do not feel bad if this collapse has caught you off guard. A lot of people have downplayed the event or never bothered to consider that the global economy would be hammered as quickly or deeply. But you need to pivot and now.

Step Six: Rally your team and get connected. Talk openly, get all scenarios on the white board (virtual for those social distancing), and acknowledge that this is going to take some significant shifting to get through it. May involve salary reductions, furloughs, or trimming staff. Better to communicate openly. Partners/principals/owners tread carefully here. How you deal with your people will impact your brand going forward. There is a lot of collaboration and rallying together across the industries and communities. Callously tossing people out to bend your cost curve will come back on you at some point. You may not be able to save everyone, and you need not fall on your sword to be noble but before you start cutting, ask what you can do to instead save your team. If they were good enough have it the firm before this, then perhaps worth going further to hold on to them?

Step Seven: Identify what your verticals need and get to work on trying to support them. Note that I did not use the term, customer. While they obviously fit into the picture, I invite you to try and provide thought leadership to the markets that you serve. Those that lead and step are those that will be most remembered.

Step Eight: Do not, under any circumstances view your salespeople as anything less than your partners and among your most precious resources. Sales are going to be impacted, accept this. Pressuring your team is short sighted, fear based and downright stupid. They want to succeed and want to win if they are worth a damn. If they are not of that caliber then shame on you for hiring the wrong people but that is another matter. You should be taking as much feedback as you can and looking to support them in any way possible.

Step Nine: Get your analysts on message. What are they key points that you need to offer your market so you can be seen as a credible thought leader? No, this is not simple and nor should it be. Lots of noise out there in the markets and people need reliable and trusted resources. Parroting conventional wisdom, pushing hopeful optimism because you need to for your own selfish reasons or getting out too quickly without having considered everything will damage your brand in ways you do not want.

Step Ten: Get engaged. Layout your go-to-market plans, refine your messages for your channels, start reaching out to clients and crank up the conversations. Be authentic, understanding and go as far as you can to provide value.

Decent people will remember how people acted throughout all of this. If you are kind and generous to your people, then they will be your ambassadors down the road. If you offered value to your clients, then they will remember that as well. The less than professional, self-serving types or those simply not up for the task at hand will be remembered as well.