“Cargo Cult” science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels

Traditional organization and information systems (IS) researchers have stridently resisted data collections using online data panels, such as Amazon's Mechanical Turk (MTurk). Although many of their concerns are legitimate, we strongly disagree with the grounds and substance of their reasons for avoiding such data collections—especially their flawed assumption that paper-based survey methods are inherently superior simply based on “tradition”, which is a highly unscientific practice we label as “cargo cult science”. To address this issue, we summarize several of the major criticisms traditionalists use against MTurk data, and we explain (1) how many of these criticisms apply more strongly to traditional survey methods, and (2) how by using advanced features of MTurk in conjunction with survey software such as Qualtrics or SurveyMonkey, researchers can navigate around many of these limitations. We conclude by demonstrating several leading practices that can be used to achieve high quality data collections with MTurk and the several advantages of doing so. Nonetheless, even when conducting traditional paper-based surveys, researchers can benefit from several (not all) of the leading methodological practices that have been developed by those who have pushed the boundaries of data collection using online panels—including for organization-level data collections. We conclude by cautioning that no “proven” method without inherent flaws exists, and organization and IS research would benefit from a clearer articulation and understanding of the range of methods and data sources available, along with their limitations and advantages.