The practice of “Just In Time” report publishing has emerged in the market research reports business over the past few years. The providers will advertise reports with titles loaded with keywords for SEO purposes and descriptions created via AI tools.  The problem is that the reports do not actually exist.

For market research buyers the product they may order is rapidly assembled from previously issued reports and updated with content sourced via web scrapers and bots. It would be one thing if providers had significant amounts of quality content and forecasting that could be drawn from to issue a customized inquiry. It is quite another when the provider’s content is of lower quality and lesser value.  The key thing to consider is the provider.  A large-scale firm with significant market presence, prestige and a track record can provide much greater value and less risk than the research factories operating offshore.

So how can you tell if the product is real or waiting to be assembled? 

A sample should be readily available, with content more substantive than boilerplate, no blanked-out pages and something that shows what might be available.  If the provider cannot show an actual sample, it likely means the report needs to be assembled.

If there are several reports available within their coverage sector(s) (often long tail) with several keywords and phrases in the titles, the provider is likely engaging in factory practices.

Tables of contents and descriptions look similar, with minor variations within sections.

The provider claims to have several hundred reports available that are all “recently updated”  Impossible given the economics and market conditions.  Sectors like PC, semiconductor or consumer electronics can be updated quarterly but that product is more obvious to the buyer.

The offer of X% customized for prospective buyers?  Factory assembly job

No name analysts listed on the reports or on the site.

If there are analysts associated with the product, how many have they produced within a certain time?  Analysts can reasonably issue 700 pages of content per year in addition to the normal research functions they engage in.  A high number of reports with 150 pages is a red flag.

Buyer Advice

While this is not an exhaustive list, it should present buyers with some things to consider.  Corporate buyers in charge of sourcing data and content owe their internal stakeholders the best decision support materials available.  Individual buyers may not be aware of what the industry practices are and should take note of the items listed above.

Stakeholders should insist on a bio of the analyst(s), talk to them for 15 minutes, ask some hard questions on availability of the product, ask them to explain how they can produce what they claim to have available in such timeframes.  Talk to the analyst to see if they have any expertise in the space as well.  Market research is fine but if they aren’t subject experts, why bother?

Market reports used to be a great way for companies to access forecasts, competitive intelligence, market feedback and analyst takes.  As market research factories have grown in scale and market presence, the value of reports has declined due to the practices employed within these shops.  Buyers are left to sift through piles of poor-quality products and do not necessarily understand the hidden risks.

A little awareness is a good thing….

New technology is where the action is. A lot of excitement, money, endless potential for innovation, disruption, and money. The perfect place for the Optimist to do their trade.

The Optimist gets paid for the sunshine they spread, not the forecasts of risk, negativity about companies’ downsides or alternatives that don’t support what they are advocating for. The analyst isn’t attached to the future outcomes in the way the Optimist is. The analyst is more concerned with their professional reputation.

There isn’t anything wrong with analysts being optimistic about market opportunities presented by new technology introductions or advancements. If there is a need, a means, and a viable business case then by all means, be bullish on the future.

But, optimists also never seem to the learn the lessons that the Gartner Hype Cycle teaches (https://www.linkedin.com/pulse/8-lessons-from-20-years-hype-cycles-michael-mullany/) They are reliable for getting on or in front of the next big thing, whatever that is and having a track record of issuing hockey stick/exponential forecasts and market outlooks that that ignore factors or historical frames of reference.

Red Flags:

Market reports or forecasts where everything grows and with a decided major ramp in 3 years and the market quintuples in value by year? If this happens over time you are dealing with an optimist. Hard pass if you are smart.

No negative comments or controversial takes or tries not offending anyone that might buy their stuff or they have bought in hard and won’t point out the risks?

When the quarter is down or there is a pattern of decline, do they step back from previous comments and offer new guidance or do they double down on the rebound that is coming? Do their comments resemble word salad to hide the bad news?

They produce vendor events that sell the dream vs the more objective reality of the market. The more optimism and future riches they profess the less you should trust them.
If they sell vendor advertising on their analyst platform, that is a clear sign they are not really in the analyst business. If the firm is aligned with a media outlet, be cautious as well, given the business considerations present. In addition to advertising, media platforms sell sponsored content. Pay to play isn’t good for independence.

On the other hand, if they offer paywall commentary for the good stuff then take that as a sign that there may be some objectivity possible. A buyer should be able to sort out what is there though.

If they offer advisory services, then the person or firm is most likely going to be an analyst as they must be able to push back on bad ideas or strategies or steer a client in a more beneficial direction. The optimist is far too risky to work with as an advisor as they only see the upside for vested interests or their having bought in on the vendor and industry press hype.

When you engage the Optimist, they tend to stumble, deflect or even gaslight to some extent when presented with contrarian views or are forced to explain their own when challenged. They will not own their hype tendencies.

The analyst, on the other hand, has a POV they can support and will adopt new thinking when presented with new evidence. They will continue to dig and look for ways to rebut their own analysis to make it more sound and provide value to clients and the broader market.

The arrival of LLMs such as ChatGPT, Bard and the ever-growing universe of AI tools that emerge daily, has stirred conversations as to their impact on industry sectors, business processes, people and more.  This article discusses how the widespread usage of the technologies will impact the market research and report space.

AI dominates the business and social media content news landscape.  Every day we see stories of new SaaS apps becoming available, advances in ChatGPT prompts, Google, Meta. Amazon, Microsoft, and others engaging in an arms race, and how Generative AI is being integrated into anything and everything.  Hard to keep up, isn’t it?

The upsides are there. Productivity gains, more output, more automation, and new use cases that are pushing the limits and driving innovation.  Tools that can search, summarize, rewrite, “analyze”, spot trends and even create reports for users. If the user understands the limitations of the tools and how they search, source and access data the risks are minimal.

One of the major keys to the ensuring best outcomes is via data observability (https://www.decube.io/post/data-observability-llm-accuracy) which is critical within within the analysis realm as the models only work when one can verify data sources, collection methods and accuracy.  Layering AI on top of a defined data reservoir makes a lot of sense.  Utilizing AI chatbots to support research and analysis is another story though.

Market research factories that make their money assembling reports using data scrapers or who engage in copy pasta are now turning to AI tools to support their “Just In Time” production model of posting reports on websites (their own or via aggregators) and backfilling the orders.  AI tools are more of a means to spitting out end products that are low quality, lacking in context and worse, inaccurate given the issue of LLM hallucinations where the ChatGPT, Bard etc. provide inaccurate results.

The Just In Time model is highly problematic as the factories will compile 200 plus page documents in under 3 days packaging together previously created forecasts based on data that could very well be AI impacted along with data and content collected, synthesized and summarized utilizing these tools.  And with the goal of having a report available for just about any subject or segment it is impossible for report factories to avoid these practices.

It also isn’t possible for a firm to produce 800 reports a year despite what they list on their website.  The economics simply don’t prove out.  A production output at that level normally requires 200 people in research, forecasting and analysis, especially if the firm sells 200-page reports.  Add in the sales, marketing, production, management and overhead and even with offshore labor benefits the company must generate nearly $8 million a year in revenue to break even.

AI tools are a way for them to shave costs while giving the illusion of extensive product offering and market coverage via a team of analysts that are not obvious to anyone.

Many companies buy reports for 3^rd party validation of market sizing, investment justifications, internal funding, M&A, or market perspectives. Given the issues with AI and the low standards employed by market research factories, how can a buyer or internal user feel confident in the content or numbers produced by factories or even know what they are getting though?

The answer…

Work with established research brands where the buyer is able understand the vendor’s database, research methodology and the analysts writing the report.  Quality firms can produce the analysts and present their processes in an obvious way, factories cannot.

Buy directly from reputable brands and avoid the factories and the resellers who carry these products.  While I am aware that some companies do screen the publishers, if the products sell then they stay in their catalogues.

If the report is not available when the buyer inquires, don’t wait 3-5 days for a scrape and compilation job. That should tell the buyer the product is a factory job.

Master the AI tools internally and not only make sure that the data and content sources are up to user standard but do your part to avoid supporting firms that lessen the value that good firms provide.