Physical properties of galaxies at z > 7 are of interest for understanding both the early phases of star formation and the process of cosmic reionization. Chemical abundance measurements offer valuable information on the integrated star formation history, and hence ionizing photon production, as well as the rapid gas accretion expected at such high redshifts. We use reported measurements of [O iii] 88 μm emission and star formation rate to estimate gas-phase oxygen abundances in five galaxies at z = 7.1-9.1 using the direct Te method. We find typical abundances = 7.9 (∼0.2 times the solar value) and an evolution of 0.9 ± 0.5 dex in oxygen abundance at fixed stellar mass from z ≃ 8 to 0. These results are compatible with theoretical predictions, albeit with large (conservative) uncertainties in both mass and metallicity. We assess both statistical and systematic uncertainties to identify promising means of improvement with the Atacama Large Millimeter/submillimeter Array (ALMA) and the James Webb Space Telescope (JWST). In particular we highlight [O iii] 52 μm as a valuable feature for robust metallicity measurements. Precision of 0.1-0.2 dex in Te-based O/H abundance can be reasonably achieved for galaxies at z ≈ 5-8 by combining [O iii] 52 μm with rest-frame optical strong lines. It will also be possible to probe gas mixing and mergers via resolved Te-based abundances on kiloparsec scales. With ALMA and JWST, direct metallicity measurements will thus be remarkably accessible in the reionization epoch.