FourierKAN
kan-gpt
| FourierKAN | kan-gpt | |
|---|---|---|
| 2 | 2 | |
| 578 | 614 | |
| - | - | |
| 5.6 | 7.4 | |
| 10 days ago | 5 days ago | |
| Python | Python | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
FourierKAN
- Fourier Kolmogorov-Arnold Networks
-
Kolmogorov-Arnold Networks
I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer :
https://github.com/GistNoesis/FourierKAN/
The core is really just a few lines.
In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of individual coordinates.
It should give an idea of Kolmogorov-Arnold networks representation power, it should probably converge easier than their spline version but spline version have less operations.
Of course, if my code doesn't work, it doesn't mean theirs doesn't.
Feel free to experiment and publish paper if you want.
kan-gpt
-
Kolmogorov-Arnold Networks
- Training script
I am currently working on training it on the WebText dataset to compare it to the original gpt2. Facing a few out-of-memory issues at the moment. Perhaps the vocab size (50257) is too large?
I'm open to contributions and would love to hear your thoughts!
https://github.com/AdityaNG/kan-gpt
What are some alternatives?
efficient-kan - An efficient pure-PyTorch implementation of Kolmogorov-Arnold Network (KAN).