We study the effect of multiple parton radiation to Higgs boson plus jet production at the LHC. The large logarithms arising from the small imbalance in the transverse momentum of the Higgs boson plus jet final state system are resummed to all orders in the expansion of the strong interaction coupling at the accuracy of Next-to-Leading Logarithm (NLL), by applying the transverse momentum dependent (TMD) factorization formalism. We show that the appropriate resummation scale should be the jet transverse momentum, rather than the partonic center of mass energy which has been normally used in the TMD resummation formalism. Furthermore, the transverse momentum distribution of the Higgs boson, particularly near the lower cut-off applied on the jet transverse momentum, can only be reliably predicted by the resummation calculation which is free of the so-called Sudakov-shoulder singularity problem, present in fixed-order calculations.