Our great sponsors
-
RWKV-LM
RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
-
SpikeGPT
Implementation of "SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks"
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Is there any relationship to the RWKV large language model?
The relationship is that SpikeGPT is inspired/is an implementation of RWKV with SNNs.
Surrogate gradients and BPTT, this is what is implemented in Norse https://github.com/Norse/Norse. It is also possible to compute exact gradients using the Eventprop algorithm.
Related posts
- [P] Norse - Deep learning with spiking neural networks (SNNs) in PyTorch
- Show HN: Deep learning with spiking neural networks (SNNs) in PyTorch
- [P] Norse - Deep learning with spiking neural networks (SNNs) in PyTorch
- Neuromorphic learning, working memory, and metaplasticity in nanowire networks
- Don't Mess with Backprop: Doubts about Biologically Plausible Deep Learning