The Design of Split Questionnaires

Instead of the heuristic randomization methods to design split questionnaires that are currently used in applied and academic research, we develop a methodology to design the split questionnaire to minimize information loss using estimates from a first wave or pilot study. Because the number of possible questionnaire designs is exponential in the number of questions, we apply the Modified Federov algorithm, using Kullback Leibler Distance as a design criterion, to find the optimal splits. We use Markov chain Monte Carlo procedures to impute missing values that result from the design. First of all, we illustrate the efficiency of the Modified Federov Algorithm on a small synthetic questionnaire, which enables the enumeration of all possible designs for comparison. Second, we compare the efficiency of split questionnaires generated with the proposed method to multiple matrix sampling, incomplete block designs, and a heuristic procedure based on principal components analysis, using synthetic and empirical Web survey data. We generate split questionnaire designs selecting either entire blocks of questions (between-block design) or sets of questions in each block (within-block design). Finally, we illustrate that due to reduced respondent burden the quality of data using split designs increases, compared to a full questionnaire in a field study.