Thoughts on Market Research Norms
On occasion, we are asked about comparison of quantitative research results to norms – average scores pulled from standardized data points, typically drawn from studies or questions of a similar type or within the same industry.
We do not compile data from our (usually custom-designed) research studies to be compiled into normative databases. We generally recommend against such comparison, as any conclusion drawn does little more than pose a statistical difference, easy to misinterpret and requiring a footnote to explain the foundation upon which it is based.
There are often vast disparities between samples used in normative studies and the sample obtained for custom, strategic studies. W5 believes in surveying a study-specific sample to address unique objectives and ensure appropriate representation of the target market. Normative studies, in contrast, tend to be based to very broad consumer audiences and therefore do not necessarily provide comparable – or actionable – results.
The level of project customization is important as well. Our studies are focused on unique research needs and category context, so it does not make sense to compare results side-by-side with data compiled from other studies. W5’s approach to questioning is different than a flatter approach enabling calculation of norms – therefore results will not be directly comparable.
As the research designs we develop for each study are built from the ground up, mapping to the strategic objectives of the study, the subsequent survey results are often difficult to compare side-by-side to any norms we can acquire.
We generally recommend against application of norms for several reasons:
Differences between survey instruments used to collect normative data and those used to collect data for a custom study
Disparities between the sample frame used in the studies compiled into norms as opposed to the sample frame constructed specifically for a custom study
The enormous range of variation in normative data
The adherence to strict integrity of question wording across surveys
The lack of integration of norms into a strategic business model
The lack of control over the methodology used for collection and synthesis of normative data
We are always willing to consider this type of comparative analysis as a part of a larger analytical plan, but these shortcomings only seem to support our recommendation of a customized research design; if comparison of quantitative data is in order, perhaps a pre/post design, or a benchmarking/tracking design would be appropriate.