About 100 km east of Rome, in the central Apennine Mountains, a critically endangered population of ∼50 brown bears live in complete isolation. Mating outside this population is prevented by several 100 km of bear-free territories. We exploited this natural experiment to better understand the gene and genomic consequences of surviving at extremely small population size. We found that brown bear populations in Europe lost connectivity since Neolithic times, when farming communities expanded and forest burning was used for land clearance. In central Italy, this resulted in a 40-fold population decline. The overall genomic impact of this decline included the complete loss of variation in the mitochondrial genome and along long stretches of the nuclear genome. Several private and deleterious amino acid changes were fixed by random drift; predicted effects include energy deficit, muscle weakness, anomalies in cranial and skeletal development, and reduced aggressiveness. Despite this extreme loss of diversity, Apennine bear genomes show nonrandom peaks of high variation, possibly maintained by balancing selection, at genomic regions significantly enriched for genes associated with immune and olfactory systems. Challenging the paradigm of increased extinction risk in small populations, we suggest that random fixation of deleterious alleles (i) can be an important driver of divergence in isolation, (ii) can be tolerated when balancing selection prevents random loss of variation at important genes, and (iii) is followed by or results directly in favorable behavioral changes.