Ultrafast control of the magnetization in ps timescales by fs laser pulses offers an attractive avenue for applications such as fast magnetic devices for logic and memory. However, ultrafast helicity-independent all-optical switching (HI-AOS) of the magnetization has thus far only been observed in Gd-based, ferrimagnetic amorphous (a-) rare earth-transition metal (a-RE-TM) systems, and a comprehensive understanding of the reversal mechanism remains elusive. Here, we report HI-AOS in ferrimagnetic a-Gd22-xTbxCo78 thin films, from x=0 to 18, and elucidate the role of Gd in HI-AOS in a-RE-TM alloys and multilayers. Increasing Tb content results in increasing perpendicular magnetic anisotropy and coercivity, without modifying magnetization density, and slower remagnetization rates and higher critical fluences for switching but still shows picosecond HI-AOS. Simulations of the atomistic spin dynamics based on the two-temperature model reproduce these results qualitatively and predict that the lower damping on the RE sublattice arising from the small spin-orbit coupling of Gd (with L=0) is instrumental for the faster dynamics and lower critical fluences of the Gd-rich alloys. Annealing a-Gd10Tb12Co78 leads to slower dynamics which we argue is due to an increase in damping. These simulations strongly indicate that accounting for element-specific damping is crucial in understanding HI-AOS phenomena. The results suggest that engineering the element-specific damping of materials can open up new classes of materials that exhibit low-energy, ultrafast HI-AOS.