SpikeGPT: 230M-parameter Spiking Neural Network trained to be a language model

• SpikeGPT

  7 693 7.2 Python

  Implementation of "SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks"

Thanks for reminding us! we just uploaded the model weight to huggingface and update the readme.md of our repo. (PS: this model only trained about 900M token on BookCorpus)

• RWKV-LM

  84 11,704 8.8 Python

  RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

They don't use transformer-style attention because it doesn't work well for SNNs. The additional time domain makes it scale with N3 instead of N2. Instead, they use Receptance Weighted Key Value - a technique introduced by Apple and better known for the open source RWKV-LM model.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project