Time in Sample: Searching for Conditioning in a Consumer Panel

By Zachary Arens and Darby Miller Steiger

The Gallup Panel is a new methodology for conducting consumer surveys and public opinion polling. The panel is used for client and internal Gallup surveys covering a variety of topics, such as customer evaluations, public opinion, and social behaviors. By recruiting panelists using traditional random digit dialing (RDD) methods, the panel represents the United States telephone-household population. But unlike regular RDD surveys, the Gallup Panel is able to survey members using a variety of modes, including telephone, mail, and web. Furthermore, the wealth of auxiliary information collected on the members permits sampling of targeted populations. Finally, a pre-recruited panel ensures that surveys can be conducted more quickly, more efficiently and more cheaply than a comparable RDD survey.

As with any new methodology, the accuracy of the Gallup Panel needs to be validated through rigorous testing before widespread adoption. One bias to which a panel may be subject is conditioning effect, sometimes called time-in-sample bias. It is broadly defined by Graham Kalton and colleagues as “the reactive effect of prior interviews on current responses” and is unique to longitudinal studies.

The U.S. Census Bureau’s Survey of Income and Program Participation (SIPP) Quality Profile distinguishes two categories of such biases. First, respondents may change their survey response behaviors as a result of earlier surveys, learning, for instance, the logical flow of questions from earlier interviews and altering their responses accordingly. Behavioral surveys are particularly prone to this effect when respondents who report engaging in a particular behavior are directed to a series of detailed follow-up questions. Respondents might deny those behaviors to avoid being asked extra questions, creating a systematic underestimate.

Second, respondents may change their normal behavior as a result of having taken a survey. For instance, one study found that some SIPP respondents learned through initial surveys about beneficial government programs that applied to them.

Since the Gallup Panel is a consumer panel, it differs from the SIPP and other studies that have been the basis for examining time-in-sample effects. Most such studies involve comparing estimates from different rotation groups at the same time period. However, the Gallup Panel is not organized into groups or waves. Panelists are recruited on an ongoing basis and remain members until they request otherwise or are kicked out after six consecutive failures to respond. Therefore, the time in sample will vary for each panelist. Additionally, the number of surveys a panelist has completed, which we call the burden, will be an alternative measure of time in sample. Arguably, the Gallup Panel’s continuous measures of the time in sample are better than the rotation group’s discrete one.

Furthermore, most previous studies on conditioning deal with the impact of the same survey administered a number of times. The questions are identical (or nearly so) from wave to wave. The Gallup Panel, on the other hand, fields surveys on a variety of topics from different clients. Rarely is the same survey administered more than once, and, if it is, it is typically not administered to the same panelists. Yet questionnaires need not be identical for conditioning effects to occur. According to Graham Kalton and colleagues, conditioning appears whenever respondents “reflect on an issue, thus changing their subsequent behavior or attitude.” As long as the issues are related, conditioning is possible.

Time-in-sample bias implies that the bias increases with the amount of time a panelist remains a member, but perhaps it can occur after a single event. Our concern was that simply the act of joining the panel conditions panelists. The recruiting process involves conversations with an interviewer, a welcome packet with introductory materials, and completion of an initial questionnaire, any of which could possibly have a systematic effect. To determine the presence of conditioning effects in the Gallup Panel, we needed to compare panel members with nonmembers.