efficient-kan
pykan
| efficient-kan | pykan | |
|---|---|---|
| 4 | 3 | |
| 3,038 | 12,425 | |
| - | - | |
| 5.6 | 9.1 | |
| 12 days ago | 7 days ago | |
| Python | Jupyter Notebook | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
efficient-kan
- FLaNK-AIM Weekly 06 May 2024
- Pure-PyTorch Implementation of Kolmogorov-Arnold Network (Kan)
-
Kolmogorov-Arnold Networks
Nice implementation I've been playing with, https://github.com/Blealtan/efficient-kan alongside @GistNoesis's.
pykan
-
Kolmogorov-Arnold Networks
Update2: got it to 100% training accuracy, 99 test accuracy with (2, 2, 2) shape.
Changes:
1. Increased the training set from 1000 to 100k samples. This solved overfitting.
2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%.
3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each activation function), then 30 steps with grid=10 (and initializing from the previous model), and then 30 steps with grid=20. This is idiomatic to KANs and covered in the Example_1_function_fitting.ipynb: https://github.com/KindXiaoming/pykan/blob/master/tutorials/...
Overall, my impressions are:
- it works!
- the reference implementation is very slow. A GPU implementation is dearly needed.
- it feels like it's a bit too non-linear and training is not as stable as it's with MLP + ReLU.
- Scaling is not guaranteed to work well. Really need to see if MNIST is possible to solve with this approach.
I will definitely keep an eye on this development.
What are some alternatives?
kan-gpt - The PyTorch implementation of Generative Pre-trained Transformers (GPTs) using Kolmogorov-Arnold Networks (KANs) for language modeling
examples - Analyze the unstructured data with Towhee, such as reverse image search, reverse video search, audio classification, question and answer systems, molecular search, etc.
VectorDBBench - A Benchmark Tool for VectorDB
threlte - 3D framework for Svelte
CML_AMP_Intelligent-QA-Chatbot-with-NiFi-Pinecone-and-Llama2 - The prototype deploys an Application in CML using a Llama2 model from Hugging Face to answer questions augmented with knowledge extracted from the website. This prototype introduces Pinecone as a database for storing vectors for semantic search.
LLaMA-Factory - Unify Efficient Fine-Tuning of 100+ LLMs
llamafile - Distribute and run LLMs with a single file.
vega-lite - A concise grammar of interactive graphics, built on Vega.
openvino_notebooks - 📚 Jupyter notebook tutorials for OpenVINO™
angle-grinder - Slice and dice logs on the command line
apexcharts.js - 📊 Interactive JavaScript Charts built on SVG
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.