The helical pitch formed by organic molecules, such as the α-helix of proteins, usually requires hydrogen bonding between chiral units and long-range positional order. It was recently found that certain liquid crystal oligomers can have a twist-bend nematic (NTB) phase with nanoscale heliconical structure without hydrogen bonding, molecular chirality or positional order. To understand the nature of this unique structure, here we present hard and resonant tender X-ray scattering studies of two novel sulfur containing dimer materials. We simultaneously measure the temperature dependences of the helical pitch and the correlation length of both the helical and positional order. In addition to an unexpected strong variation of the pitch with the length of the spacer connecting the monomer units, we find that at the transition to the NTB phase the positional correlation length drops. The helical structure was found not only in the NTB phase but observed even in the upper range of a smectic phase that forms just below the NTB state. The coexistence of smectic layering and the heliconical order indicates a layered (SmATB) phase wherein the rigid units of the dimers are tilted with respect to the smectic layer normal in order to accommodate the bent conformation of the dimers and the tilt direction rotates along the heliconical axis.