Why 2D Rotations Solved Transformer Long-Context: A Mechanics-First Look at RoPE Attention requires two things from positional encodings: Relative Distance Sensitivity: Token i attending to Token j should care primarily about how far apart they are ( i − j ), not where they sit globally in the context window. Feature Preservation: Injecting position information must not destroy or mangle the semantic embedding ...