Chandra data in the COSMOS, AEGIS-XD and 4 Ms Chandra Deep Field South are combined with multiwavelength photometry available in those fields to determine the rest-frame U-V versus V-J colours of X-ray AGN hosts in the redshift intervals 0.1 < z < 0.6 (mean z = 0.40) and 0.6 < z < 1.2 (mean z = 0.85). This combination of colours provides an effective and least model-dependent means of separating quiescent from star-forming, including dust reddened, galaxies. Morphological information emphasizes differences between AGN populations split by their U-V versus V-J colours. AGN in quiescent galaxies consist almost exclusively of bulges, while star-forming hosts are equally split between early-and late-type hosts. The position of AGN hosts on the U-V versus V-J diagram is then used to set limits on the accretion density of the Universe associated with evolved and star-forming systems independent of dust induced biases. It is found that most of the black hole growth at z ̃0.40 and 0.85 is associated with star-forming hosts. Nevertheless, a non-negligible fraction of the X-ray luminosity density, about 15-20 per cent, at both z = 0.40 and 0.85, is taking place in galaxies in the quiescent region of the U-V versus V-J diagram. For the low-redshift sub-sample, 0.1 < z < 0.6, we also find tentative evidence, significant at the 2σ level, that AGN split by their U-V and V-J colours have different Eddington ratio distributions. AGN in blue star-forming hosts dominate at relatively high Eddington ratios. In contrast, AGN in red quiescent hosts become increasingly important as a fraction of the total population towards low Eddington ratios. At higher redshift,z > 0.6, such differences are significant at the 2σ level only for sources with Eddington ratios ≥10-3. These findings are consistent with scenarios in which diverse accretion modes are responsible for the build-up of supermassive black holes at the centres of galaxies.We compare these results with the predictions of the GALFORM semi-analytic model for the cosmological evolution of AGN and galaxies. This model postulates two black hole fuelling modes, the first is linked to star formation events and the second takes place in passive galaxies. GALFORM predicts that a substantial fraction of the black hole growth at z < 1 is associated with quiescent galaxies, in apparent conflict with the observations. Relaxing the strong assumption of the model that passive AGN hosts have zero star formation rate could bring those predictions in better agreement with the data. © 2014 The Authors Published by Oxford University Press on behalf of the Royal Astronomical Society.