We have performed high-resolution neutron diffraction and inelastic neutron scattering experiments in the frustrated multiferroic hexagonal compounds RMnO₃ (R=Ho,Yb,Sc,Y), which provide evidence of a strong magnetoelastic coupling in the whole family. We can correlate the atomic positions, the type of magnetic structure, and the nature of the spin waves whatever the R ion and temperature. The key parameter is the position of the Mn ions in the unit cell with respect to a critical threshold of 1/3, which determines the sign of the coupling between Mn triangular planes.