Feedback request: How should I compare direct vs. conceptual replications in meta-analysis?

Are conceptual replications truly superior to direct replications? I don't think so. I am planning to do a p-curve comparison of the two. What else should I be asking to help settle this debate? Please comment below about what you would like to learn about the differences. Or you can comment on Twitter and be sure to "@" me (@aaroncharlton). 

Background 

While other social sciences have pushed for more preregistered direct replications, marketing editors who are open to accepting replications have shown a strong preference for conceptual replications (Lynch et al. 2015). As far as I have been able to ascertain through extensive research and asking on Twitter, only ONE preregistered direct replication has ever been published in a marketing journal (good job Burak and Evrim!) (Tunca and Yanar 2020). While marketing has an abysmal record for preregistered direct replications (1/14 unambiguously successful), the conceptual replications have mostly worked (Lynch et al. 2015). 

Direct replications. The advantage of the preregistered direct replication is it can help show if the original finding was even true. This could be important information given a research culture that is more inclined toward data exploration than what is indicated in published papers (Janiszewski and van Osselaer 2021).

Conceptual replications. Conceptual replications can help extend the findings into new contexts and identify boundary conditions. 

Data collection plan

This fall (Fall 2021), my lab (M&M Lab) is planning to begin manually harvesting test statistics and relevant meta-information from replications in marketing. 

  • Direct replications. Identification of preregistered direct replications of marketing journals is ongoing at https://www.openmktg.org/research/replications. All preregistered direct replications of studies that originally appeared in marketing journals will be used from all years up to and including 2021 (current N = 14). 
  • Conceptual replications. All conceptual replications from the Replication Corner up to and including 2021 will be used (N is unknown but I believe it is close to 60). The Replication Corner started at IRJM, went to JMB and is now at Marketing Letters. 

Research question

Which form of replication provides greater evidential value? This can be evaluated by harvesting p-values and plotting them in a p-curve (Simonsohn et al. 2014) and z-curve (Bartos and Schimmack 2020).  Consistent with p-curve recommendations we would only harvest the core hypothesis test statistics and disregard other tests such as manipulation checks. We will also compare sample size, effect size and statistical power. My hypothesis is that preregistered direct replications would win. I say this because I expect that conceptual replications are more likely to suffer from too many researcher degrees of freedom and confirmation bias.


References:

BartoŇ°, F., & Schimmack, U. (2020, January 10). Z-Curve.2.0: Estimating Replication Rates and Discovery Rates. https://doi.org/10.31234/osf.io/urgtn

Janiszewski, C., & van Osselaer, S. M. J. (2021). The Benefits of Candidly Reporting Consumer Research, Journal of Consumer Psychology, forthcoming.

Lynch Jr, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333-342.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: a key to the file-drawer. Journal of experimental psychology: General, 143(2), 534.

Tunca, B., & Yanar, E. (2020). Women's Luxury Products as Signals to Other Women. Journal of Marketing Behavior, 4(2-4), 227-238. https://doi.org/10.1561/107.00000066

Comments

  1. I realized it doesn't make sense to p-curve direct replications since there has only been one that has worked in marketing so far. I will probably still p-curve the conceptual replications though.

    ReplyDelete
  2. I've just preregistered it. Data collection for a p-curve of conceptual replications will start this fall: https://aspredicted.org/9qu6u.pdf

    ReplyDelete

Post a Comment

Popular posts from this blog

Conflicts between Dan Ariely's statement and Footnote #14 (DataColada #98)

RETRACTED ARTICLE: Why money meanings matter in decisions to donate time and money

The salvaging of a Ping Dong paper at Journal of Marketing