We study out-of-time ordered four-point functions in two dimensional conformal field theories by suitably analytically continuing the Euclidean correlator. For large central charge theories with a sparse spectrum, chaotic dynamics is revealed in an exponential decay; this is seen directly in the contribution of the vacuum block to the correlation function. However, contributions from individual non-vacuum blocks with large spin and small twist dominate over the vacuum block. We argue, based on holographic intuition, that suitable summations over such intermediate states in the block decomposition of the correlator should be sub-dominant, and attempt to use this criterion to constrain the OPE data with partial success. Along the way we also discuss the relation between the spinning Virasoro blocks and the on-shell worldline action of spinning particles in an asymptotically AdS spacetime.