Evaluating sample allocation and effort in detecting population differentiation for discrete and continuously distributed individuals

One of the most pressing issues in spatial genetics concerns sampling. Traditionally, substructure and gene flow are estimated for individuals sampled within discrete populations. Because many species may be continuously distributed across a landscape without discrete boundaries, understanding sampling issues becomes paramount. Given large-scale, geographically broad conservation efforts, researchers are looking for guidance as to the trade-offs between sampling more individuals within a population versus few individuals scattered across more populations. Here, we conducted simulations that address these issues. We first established two archetypical patterns of dispersion: (1) individuals within discrete populations, and (2) continuously distributed individuals with limited dispersal. We used genotypes generated from a spatially-explicit, individual-based program and simulated genetic structure in individuals from nine different population sizes across a landscape that either had barriers to movement (defining discrete populations) or isolation-by-distance patterns (defining continuously distributed individuals). Then, given each pattern of dispersion, we allocated samples across four different sampling strategies for each of the nine population sizes in various configurations for sampling more individuals within a population versus fewer individuals scattered across more populations. We assessed the population genetic substructure with both the population-based metric, F_ST, and an individual-based metric, D_PS regardless of the true pattern of dispersion to allow us to better understand the effect of incorrectly matching the metric and the distribution (e.g., F_ST with continuously distributed individuals, and vice versa). We show that sampling many subpopulations (or sampling areas), thus sampling fewer individuals per subpopulation, overestimates measures of population subdivision with the population-based metric for both patterns of dispersion. In contrast, using the individual-based metric gives the opposite results: sampling too few subpopulations, and many individuals per subpopulation, produces an underestimate of the strength of isolation-by-distance. By comparing all results, we were able to suggest a strong predictive model of a chosen genetic structure metric for elucidating the sampling design trade-offs given each pattern of dispersion and configuration on the landscape.