LongNet VS long_llama

i wish they would implement this into character.ai github.com/kyegomez/LongNet

If you want to talk immature looking, longnet wouldn't even compile. That's a big oof, considering it's a python and usually nonworking code is good enough to generate byte code. (also it has hard-coded dtype and device)

When will the Open Source LLMs start using LongNet https://github.com/kyegomez/LongNet https://arxiv.org/abs/2307.02486

same time we also have chart like this, that shows that models with extended context work well with contexts longer they are learnt on https://github.com/CStanKonrad/long_llama

LONGLLAMA : extending LLaMA’s context length with FOT One of the promises of our work is that FOT can be used to fine-tune already existing large models to extend their context length. In this section, we show that this is indeed the case. We use OpenLLaMA-3B and OpenLLaMA-7B models trained for 1T tokens as start- ing points and fine-tune them with FOT. We show that the resulting models, which we call LONGLLAMAs, are capable of extrapolating beyond their training context length (even up to 256K) and retain the performance on short-context tasks. We release the inference code on GitHub: https://github.com/CStanKonrad/long_llama and the LONGLLAMA-3B check- point on Hugging Face: https://huggingface.co/syzymon/long_llama_3b. We note that our checkpoint is backward compatible, i.e. can be used with any existing LLaMA inference code (both in Hugging Face and other implementations), albeit without long-context capabilities