Feedback request: How should I compare direct vs. conceptual replications in meta-analysis?
Are conceptual replications truly superior to direct replications? I don't think so. I am planning to do a p-curve comparison of the two. What else should I be asking to help settle this debate? Please comment below about what you would like to learn about the differences. Or you can comment on Twitter and be sure to "@" me (@aaroncharlton).
While other social sciences have pushed for more preregistered direct replications, marketing editors who are open to accepting replications have shown a strong preference for conceptual replications (Lynch et al. 2015). As far as I have been able to ascertain through extensive research and asking on Twitter, only ONE preregistered direct replication has ever been published in a marketing journal (good job Burak and Evrim!) (Tunca and Yanar 2020). While marketing has an abysmal record for preregistered direct replications (1/14 unambiguously successful), the conceptual replications have mostly worked (Lynch et al. 2015).
Direct replications. The advantage of the preregistered direct replication is it can help show if the original finding was even true. This could be important information given a research culture that is more inclined toward data exploration than what is indicated in published papers (Janiszewski and van Osselaer 2021).
Conceptual replications. Conceptual replications can help extend the findings into new contexts and identify boundary conditions.
Data collection plan
This fall (Fall 2021), my lab (M&M Lab) is planning to begin manually harvesting test statistics and relevant meta-information from replications in marketing.
- Direct replications. Identification of preregistered direct replications of marketing journals is ongoing at https://www.openmktg.org/research/replications. All preregistered direct replications of studies that originally appeared in marketing journals will be used from all years up to and including 2021 (current N = 14).
- Conceptual replications. All conceptual replications from the Replication Corner up to and including 2021 will be used (N is unknown but I believe it is close to 60). The Replication Corner started at IRJM, went to JMB and is now at Marketing Letters.
Which form of replication provides greater evidential value? This can be evaluated by harvesting p-values and plotting them in a p-curve (Simonsohn et al. 2014) and z-curve (Bartos and Schimmack 2020). Consistent with p-curve recommendations we would only harvest the core hypothesis test statistics and disregard other tests such as manipulation checks. We will also compare sample size, effect size and statistical power. My hypothesis is that preregistered direct replications would win. I say this because I expect that conceptual replications are more likely to suffer from too many researcher degrees of freedom and confirmation bias.
Bartoš, F., & Schimmack, U. (2020, January 10). Z-Curve.2.0: Estimating Replication Rates and Discovery Rates. https://doi.org/10.31234/osf.io/urgtn
Janiszewski, C., & van Osselaer, S. M. J. (2021). The Benefits of Candidly Reporting Consumer Research, Journal of Consumer Psychology, forthcoming.
Lynch Jr, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333-342.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of experimental psychology: General, 143(2), 534.
Tunca, B., & Yanar, E. (2020). Women's Luxury Products as Signals to Other Women. Journal of Marketing Behavior, 4(2-4), 227-238. https://doi.org/10.1561/107.00000066