[R] RWKV-v2-RNN : A parallelizable RNN with transformer-level LM performance, and without using attention

1. RWKV-LM

   1 85 13,346 9.0 Python

   RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

   

   Simply run train.py in https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v2-RNN :)

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. AI-Writer

   2 2 3,106 3.4 Python

   AI 写小说，生成玄幻和言情网文等等。中文预训练生成模型。采用我的 RWKV 模型，类似 GPT-2 。AI写作。RWKV for Chinese novel generation.

   

   I need more FLOPS lol. On the other hand, quite some users have fine-tuned the Chinese novel model (https://github.com/BlinkDL/AI-Writer).

4. RWKV-v2-RNN-Pile

   3 6 67 0.0 Python

   RWKV-v2-RNN trained on the Pile. See https://github.com/BlinkDL/RWKV-LM for details.

   

   Yes. You can begin with the 169M params model (in Releases of https://github.com/BlinkDL/RWKV-v2-RNN-Pile) which is not converged yet but fine for testing.

5. SmallInitEmb

   4 1 59 3.2 Python

   LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence

   

   SmallInitEmb (https://github.com/BlinkDL/SmallInitEmb)

6. RWKV-CUDA

   5 3 220 2.9 Cuda

   The CUDA version of the RWKV language model ( https://github.com/BlinkDL/RWKV-LM )

   

   It's using my custom CUDA kernel ( https://github.com/BlinkDL/RWKV-CUDA ) to speedup training, so only GPU for now. On the other hand, you don't need CUDA for inference, and it is very fast even on CPUs.

7. token-shift-gpt

   6 1 47 0.0 Python

   Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing

   

   indeed :) took this to the extreme with https://github.com/lucidrains/token-shift-gpt

Suggest a related project