Architecture

Rotary Position Embedding (RoPE)

RoPE applies rotations proportional to position to query and key vectors. It encodes absolute position through rotation and naturally captures relative position through rotation composition. RoPE shows better generalization to longer sequences than absolute positional encoding. It's used in many modern models. RoPE is simpler than some alternatives and works well in practice. It enables models to generalize somewhat beyond training length. Theoretical properties and empirical performance are both strong.