[R] Mega: Moving Average Equipped Gated Attention. By using LSTM-style gates, Mega outperforms Transformer and S4 over Long Range Area, NMT, ImageNet, Wikitext-103 and raw speech classification.

• RWKV-LM

  84 11,747 8.8 Python

  RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

thanks for the paper and the great experimental results! the findings line up with the "time delay" use in https://github.com/BlinkDL/RWKV-LM from /u/bo_peng

• fairseq-apollo

  2 104 8.7 Python

  FairSeq repo with Apollo optimizer

Here is the code https://github.com/XuezheMax/fairseq-apollo The checkpoints are releasing soon.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project