micrograd
NNfSiX
Our great sponsors
micrograd | NNfSiX | |
---|---|---|
22 | 46 | |
8,273 | 1,348 | |
- | - | |
0.0 | 0.0 | |
5 days ago | 7 months ago | |
Jupyter Notebook | C++ | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
micrograd
-
Micrograd-CUDA: adapting Karpathy's tiny autodiff engine for GPU acceleration
I recently decided to turbo-teach myself basic cuda with a proper project. I really enjoyed Karpathy’s micrograd (https://github.com/karpathy/micrograd), so I extended it with cuda kernels and 2D tensor logic. It’s a bit longer than the original project, but it’s still very readable for anyone wanting to quickly learn about gpu acceleration in practice.
-
Stuff we figured out about AI in 2023
FOr inference, less than 1KLOC of pure, dependency-free C is enough (if you include the tokenizer and command line parsing)[1]. This was a non-obvious fact for me, in principle, you could run a modern LLM 20 years ago with just 1000 lines of code, assuming you're fine with things potentially taking days to run of course.
Training wouldn't be that much harder, Micrograd[2] is 200LOC of pure Python, 1000 lines would probably be enough for training an (extremely slow) LLM. By "extremely slow", I mean that a training run that normally takes hours could probably take dozens of years, but the results would, in principle, be the same.
If you were writing in C instead of Python and used something like Llama CPP's optimization tricks, you could probably get somewhat acceptable training performance in 2 or 3 KLOC. You'd still be off by one or two orders of magnitude when compared to a GPU cluster, but a lot better than naive, loopy Python.
[1] https://github.com/karpathy/llama2.c
[2] https://github.com/karpathy/micrograd
-
Writing a C compiler in 500 lines of Python
Perhaps they were thinking of https://github.com/karpathy/micrograd
- Linear Algebra for Programmers
- Understanding Automatic Differentiation in 30 lines of Python
-
Newbie question: Is there overloading of Haskell function signature?
I was (for fun) trying to recreate micrograd in Haskell. The ideia is simple:
-
[D] Backpropagation is not just the chain-rule, then what is it?
Check out this repo I found a few years back when I was looking into understanding pytorch better. It's basically a super tiny autodiff library that only works on scalars. The whole repo is under 200 lines of code, so you can pull up pycharm or whatever and step through the code and see how it all comes together. Or... you know. Just read it, it's not super complicated.
-
Neural Networks: Zero to Hero
I'm doing an ML apprenticeship [1] these weeks and Karpathy's videos are part of it. We've been deep down into them. I found them excellent. All concepts he illustrates are crystal clear in his mind (even though they are complicated concepts themselves) and that shows in his explanations.
Also, the way he builds up everything is magnificent. Starting from basic python classes, to derivatives and gradient descent, to micrograd [2] and then from a bigram counting model [3] to makemore [4] and nanoGPT [5]
[1]: https://www.foundersandcoders.com/ml
[2]: https://github.com/karpathy/micrograd
[3]: https://github.com/karpathy/randomfun/blob/master/lectures/m...
[4]: https://github.com/karpathy/makemore
[5]: https://github.com/karpathy/nanoGPT
-
Rustygrad - A tiny Autograd engine inspired by micrograd
Just published my first crate, rustygrad, a Rust implementation of Andrej Karpathy's micrograd!
-
Hey Rustaceans! Got a question? Ask here (10/2023)!
I've been trying to reimplement Karpathy's micrograd library in rust as a fun side project.
NNfSiX
-
Are there any books I should read to learn machine learning from scratch?
I've been rather enjoying "Neural Networks from Scratch" (https://nnfs.io/)
-
Ask HN: Those learning about neural networks, what do you find most difficult?
I haven't gotten super deep into it yet, but https://nnfs.io/ has been good in my opinion. The book slowly replaces written and explained code with numpy equivalents to keep the examples fast. Plus the accompanying animations are also useful. I would be curious what others think on it too.
- Gutes Einführungsbuch zu KI
- [Deep Learning] Neural Networks from Scratch in Python
- What do I get a programming obsessed high school boy for his birthday? I actually need advice
-
GPT in 60 Lines of NumPy
For those curious to writing "gradient descent with respect to some loss function" starting from an empty .py file (and a numpy import, sure), can't recommend enough Harrison "sentdex" Kinsley's videos/book Neural Networks from Scratch in Python [1].
[1] https://youtu.be/Wo5dMEP_BbI?list=PLQVvvaa0QuDcjD5BAw2DxE6OF... https://nnfs.io
-
Ask HN: What are the foundational texts for learning about AI/ML/NN?
Not sure if foundational (quite a tall order in such a fast-moving field), but for sure a nice introduction into neural networks, and even mathematics in general (because it's nice to see numbers in action beyond school-level algebra):
Harrison Kinsley, Daniel Kukiela, Neural Networks from Scratch, https://nnfs.io, https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0Qu...
-
Ask HN: How to get back into AI?
Have you had a look at https://nnfs.io/ ? I bought the book and am gearing up to start working through it, I would be interested to know your thoughts. Generally I want to chart a personal curriculum from data engineer to practical application of modern AI to real business problems.
-
Programming an AI as a beginner
You can check out Neural Networks from Scratch in Python for an introduction to neural networks, which can be used for image classification. Please be forewarned that you'll need the mathematics necessary to read through this book - however, I'm assuming that since you've selected writing such an algorithm(s) in Python for your final school project that you're aware of such.
-
Moved to amd today and holy it's amazing
I am planning on working my way through Neural Networks From Scratch (https://nnfs.io/) in a few months just to build my understanding. After that I'm hoping to be able to figure out the best path for a couple of projects I have in mind.
What are some alternatives?
deepnet - Educational deep learning library in plain Numpy.
deeplearning-notes - Notes for Deep Learning Specialization Courses led by Andrew Ng.
tinygrad - You like pytorch? You like micrograd? You love tinygrad! ❤️ [Moved to: https://github.com/tinygrad/tinygrad]
ML-From-Scratch - Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.
minGPT - A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training
yolov7 - Implementation of paper - YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
ProjectOne - The project is to build a neural network from scratch. The motivation for this project is from nnfs.io a website build by @Sentdex. Nnfs.io is actually meant for a book that teaches the fundamentals of neural network and help us to build our own network. Let's build a new neural network where we can learn the fundamentals and make a great hands-on work space for aspiring machine learning engineers and the GitHub community
machine.academy - Neural Network training library in C++ and C# with GPU acceleration
best-of-ml-python - 🏆 A ranked list of awesome machine learning Python libraries. Updated weekly.