The evolution of the antiferromagnetic phase across the temperature-driven ferromagnetic (FM) to antiferromagnetic (AF) phase transition in epitaxial FeRh thin films was studied by x-ray magnetic linear and circular dichroism (XMLD and XMCD) and photoemission electron microscopy. By comparing XMLD and XMCD images recorded at the same temperature, the AF phase was identified, its structure directly imaged, and its evolution studied across the transition. A quantitative analysis of the correlation length of the images shows differences between the characteristic length scale of the two phases with the AF phase having a finer feature size. The asymmetry of the transition from FM to AF upon cooling and AF-FM upon heating is evidenced: upon cooling the formation of AF phase is dominated by nucleation at defects, with little subsequent growth, resulting in a small and non-random final AF domain structure, while upon heating, heterogeneous nucleation at different sites followed by significant domain size growth of the FM phase is observed, resulting in a non-reproducible final FM large domain structure.