Using simulations to evaluate Mantel-based methods for assessing landscape resistance to gene flow

Katherine A. Zeller, Tyler G. Creech, Katie L. Millette, Rachel S. Crowhurst, Robert A. Long, Helene H. Wagner, Niko Balkenhol, Erin L. Landguth

Research output: Contribution to journalArticlepeer-review

43 Scopus citations


Mantel-based tests have been the primary analytical methods for understanding how landscape features influence observed spatial genetic structure. Simulation studies examining Mantel-based approaches have highlighted major challenges associated with the use of such tests and fueled debate on when the Mantel test is appropriate for landscape genetics studies. We aim to provide some clarity in this debate using spatially explicit, individual-based, genetic simulations to examine the effects of the following on the performance of Mantel-based methods: (1) landscape configuration, (2) spatial genetic nonequilibrium, (3) nonlinear relationships between genetic and cost distances, and (4) correlation among cost distances derived from competing resistance models. Under most conditions, Mantel-based methods performed poorly. Causal modeling identified the true model only 22% of the time. Using relative support and simple Mantel r values boosted performance to approximately 50%. Across all methods, performance increased when landscapes were more fragmented, spatial genetic equilibrium was reached, and the relationship between cost distance and genetic distance was linearized. Performance depended on cost distance correlations among resistance models rather than cell-wise resistance correlations. Given these results, we suggest that the use of Mantel tests with linearized relationships is appropriate for discriminating among resistance models that have cost distance correlations <0.85 with each other for causal modeling, or <0.95 for relative support or simple Mantel r. Because most alternative parameterizations of resistance for the same landscape variable will result in highly correlated cost distances, the use of Mantel test-based methods to fine-tune resistance values will often not be effective.

Original languageEnglish
Pages (from-to)4115-4128
Number of pages14
JournalEcology and Evolution
Issue number12
StatePublished - Jun 1 2016


  • landscape fragmentation
  • landscape genetics
  • landscape resistance
  • simulations


Dive into the research topics of 'Using simulations to evaluate Mantel-based methods for assessing landscape resistance to gene flow'. Together they form a unique fingerprint.

Cite this