Gene flow among populations can enhance local adaptation if it introduces new genetic variants available for selection, but strong gene flow can also stall adaptation by swamping locally beneficial genes. These outcomes can depend on population size, genetic variation, and the environmental context. Gene flow patterns may align with geographic distance (IBD--isolation by distance), whereby immigration rates are inversely proportional to the distance between populations. Alternatively gene flow may follow patterns of isolation by environment (IBE), whereby gene flow rates are higher among similar environments. Finally, gene flow may be highest among dissimilar environments (counter-gradient gene flow), the classic "gene-swamping" scenario. Here we survey relevant studies to determine the prevalence of each pattern across environmental gradients. Of 70 studies, we found evidence of IBD in 20.0%, IBE in 37.1%, and both patterns in 37.1%. In addition, 10.0% of studies exhibited counter-gradient gene flow. In total, 74.3% showed significant IBE patterns. This predominant IBE pattern of gene flow may have arisen directly through natural selection or reflect other adaptive and nonadaptive processes leading to nonrandom gene flow. It also precludes gene swamping as a widespread phenomenon. Implications for evolutionary processes and management under rapidly changing environments (e.g., climate change) are discussed.