Due to Earth's revolution around the Sun, the expected scattering rate in direct dark matter searches is annually modulated. This modulation is expected to differ between experiments when given as a function of recoil energy ER, e.g. due to the gravitational focusing effect of the Sun. A better variable to compare results among experiments employing different targets is the minimum speed vmin a dark matter particle must have to impart a recoil energy ER to a target nucleus. It is widely believed that the modulation expressed as a function of vmin is common to all experiments, irrespective of the dark matter distribution. We point out that the annual modulation as a function of vmin, and in particular the times at which the rate is maximum and minimum, could be very different depending on the detector material. This would be an indication of a scattering cross section with nonfactorizable velocity and target material dependence. Observing an annual modulation with at least two different target elements would be necessary to identify this type of cross section.