We present a detailed quantum oscillation study of the Fermi surface of the recently discovered Yb-based heavy fermion superconductor beta-YbAlB4. We compare the data, obtained at fields from 10 to 45 T, to band structure calculations performed using the local density approximation. Analysis of the data suggests that f holes participate in the Fermi surface up to the highest magnetic fields studied. We comment on the significance of these findings for the unconventional superconducting properties of this material.