The bromodomain protein Brd4 promotes HIV-1 latency by competitively inhibiting P-TEFb-mediated transcription induced by the virus-encoded Tat protein. Brd4 is recruited to the HIV LTR by interactions with acetyl-histones3 (AcH3) and AcH4. However, the precise modification pattern that it reads and the writer for generating this pattern are unknown. By examining a pool of latently infected proviruses with diverse integration sites, we found that the LTR characteristically has low AcH3 but high AcH4 content. This unusual acetylation profile attracts Brd4 to suppress the interaction of Tat with the host super elongation complex (SEC) that is essential for productive HIV transcription and latency reversal. KAT5 (lysine acetyltransferase 5), but not its paralogs KAT7 and KAT8, is found to promote HIV latency through acetylating H4 on the provirus. Antagonizing KAT5 removes AcH4 and Brd4 from the LTR, enhances the SEC loading, and reverses as well as delays, the establishment of latency. The pro-latency effect of KAT5 is confirmed in a primary CD4+ T cell latency model as well as cells from ART-treated patients. Our data thus indicate the KAT5-AcH4-Brd4 axis as a key regulator of latency and a potential therapeutic target to reactivate latent HIV reservoirs for eradication.