There is no better time to collect baseline data than today. As panels proliferate they represent shifting challenges in transition. In this new ground research, we have collected and continue to collect data on an unprecedented basis; to create a base-line to track, clarify and project the panel universe as it changes. We have translated the 19 minute questionnaire that we employed in our analysis of the American panels (CASRO February 2009) into the local languages required by 40 international markets. Panel companies around the world have agreed to provide us sample in those forty markets. Before summer of 2009 some 100+ waves, of 400 interviews each, will be completed globally. Our goal is to obtain data from as many sources as possible in the forty target markets.
Our focus in the American markets was inter-panel and sourcing mode differences in buying behavior (Gittelman and Trimarchi, 2009). We shall carry this analysis forward globally. We found that within sourcing modes considerable homogeneity existed with variance most predominant between modes. Our preliminary analysis shows that homogeneity is far more widespread globally as many of the respondent groups that are driving change have not as yet increased their ranks. There are definitive patterns that allow us to index demographic, psychographic, sociologic and buyer behavior. Through these analyses we are able to discern norms in data collection sources, measure differences in key variables, and create models for comparing between and within international boundaries as well as demonstrate variance between sources.
Homogeneity is a key element in online sample health. In a non probabilistic sampling world where we live without a net, multiple mode sampling is in vogue. But the prospect of combining disparate groups without understanding key behavioral differences is an unhealthy one. Even if we achieve an understanding of where panels stand, we have no ability to quantify and project change unless we establish a baseline study.
Survey research has historically relied on a probabilistic model to underlie its sampling frame. With rare exception online research is non-probabilistic. Research without the safety net of a probabilistic frame raises all kinds of alarms. Challenges as to the reliability of online research has become a growing crescendo as its non-probabilistic nature has become evident. However, not all sampling frames must be probabilistic. To insure that sampling frames exhibit appropriate levels of continuity, predictability and reliability they must be measured over time. Unfortunately, no such standard metrics exist to track reliability in online sampling. In fact, whether they are access panels or social networks there are no standardized means of balancing panels or even comparing them.
We have developed a family of sampling standards based upon segmentation by key variables such as, but not limited, to media, psychographics and in particular buying behavior. It is the latter that we view as most critical to our clients. From these metrics we are beginning the process of measuring consistency over time in panels. A number of panels globally have asked us to track their progress through a series of continuity waves and analyze their variability as they move through time. It is our intent to provide the results of these models to firms who seek a measure of their consistency.
In addition, to measuring performance, we believe that there are three key requirements for standard panel metrics including: (1) the ability to capture panel performance variations consistent with the differing needs of sample users, (i.e. a broadcasting company might wish to anchor its sampling frame to media segments); (2) The ability to create a data base that is prospective in that new sample sources can be added to the database without repeating the analysis and (3) a focus on indices that are pragmatic in their measure (i.e. We always view buying behavior as the most pragmatic.)
We propose to use segmentation analysis as a new metric that will allow us to anchor online data in a new non probabilistic sampling frame. It is the existence of the global data that gives us a rare opportunity to experiment with this new methodology. Our goal is to use segmentation in each country to create a fingerprint that can be consistently maintained by blending panels. By minimizing the variability from the segments through optimization and panel combination we will establish a means for stabilizing online data irrespective of the sourcing modes from which they draw their origin. We cannot stabilize online data unless we provide it with a reference point to anchor itself; the segments are that anchor. As the sourcing models continue to shift, panels will age and shift with them; we need a reliable anchor that rises above these problems. It is essential that we explore tools to measure these changes. Without a means of comparison we cannot expect to measure drift nor can we expect to have a platform for predicting the future. We do not profess to be on the road to a new probabilistic framework but rather a platform for comparison and continuity. Our effort is to provide a continuing platform that provides a stable reference point.