More Rope Scaling Implementations (PI, Yarn)

[tyler-romero]

Implements

1. Position Interpolation
2. Stepwise (llama 3.1) scaling
3. Yarn

[abertsch72]

thanks so much for standardizing this! were you also going to add a way to turn on/off sliding-window-layer scaling? this seems more important now that it seems like some of our sliding window scaling runs actually outperform global-only scaling

[abertsch72]

do we have to restrict attention rescaling factor to be >1?

[abertsch72]

nit, but the name suggests to me that you get more finegrained control than high/low...

[tyler-romero]

Yeah good point, I'll switch to calling it stepwise

[abertsch72]

it would be nice if these params used the same name / interpretation as the YaRN params that control the same thing... minimally, I'd vote for calling them something other than factor because we also have factor above that does a different thing

(I know these names are probably holdovers from the methods, though-- if this is how huggingface does the two methods then maybe it's better to be consistent with them rather than internally consistent)

[dirkgr]

The way you preserve backwards compatibility here is by having the base class do basic RoPE extension like the RoPE paper says, and have subclasses that do the fancier stuff?

[tyler-romero]

PerFrequencyRoPEScalingConfig is the same as the old rope scaling config. I think its wrong to think of that as the base case, and there are 0 dependencies on that in main, so I'm not sure its worth distorting the factorization of the code for.