|over 2 years ago||22 days ago|
|MIT License||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
I Still ‘Lisp’ (and You Should Too)
4 projects | news.ycombinator.com | 20 Feb 2023
Well, I wrote a genetic programming library, and it was fun to parse a Lisp-like representation from Python. You still have recursion and everything (albeit no tail call optimization).
Here, `_from_source` goes from a plain array of tokens to a nested one (tree), depending on their arity:
Lisp is almost valid Python. The exception is the single-element tuple which needs a comma: (x,)
But I still preferred to use Python as a programming language, and Lisp as a sort of AST. It's just easier. I am curious what roadblocks you faced in your ASCII delimited parsing.
Do you by any chance still have the two parsers? I'd love to see them. If you are worried about your anonymity, you can find my website on my HN profile, and my e-mail on my website. I promise not to disclose your identity publicly.
Do Simpler Machine Learning Models Exist and How Can We Find Them?
5 projects | news.ycombinator.com | 22 Dec 2022
If interpretability is sufficiently important, you could straight-up search for mathematical formulae.
My SymReg library pops to mind. I'm thinking of rewriting it in multithreaded Julia this holiday season.
I made an Entity Component System
2 projects | /r/Python | 15 Jun 2022
Indeed, I ran face-first into Python's GIL that prevents any useful CPU-bound multithreading, with my symbolic regression library.
Show HN: 78% MNIST accuracy using GZIP in under 10 lines of code
5 projects | news.ycombinator.com | 20 Sep 2023
If you'd like to play around with MNIST yourself, I wrote a PyTorch training implementation that gets ~95.45%+ in <13.6 seconds on a V100, est. < 6.5 seconds on an A100. Made to be edited/run in Colab: https://github.com/tysam-code/hlb-CIFAR10
It's originally kitted for CIFAR10, but I've found the parameters to be quite general. The code is very easy to read and well-commented, and is a great starting place for exploration.
Min-cut deltas to run MNIST:
`.datasets.CIFAR10('` -> `.datasets.MNIST('` (both occurences)
The Mathematics of Training LLMs
3 projects | news.ycombinator.com | 16 Aug 2023
Sure. Basically everything in https://github.com/tysam-code/hlb-CIFAR10 was directly founded on the concepts in the paper, down to the coding, commenting, and layout styles (hence why I advocate so strongly for it as a requirement for ML. The empirical benefits are clear to me).
Before I sat down and wrote my first line, I spent a very long time thinking about how to optimize the repo. Not just in terms of information flow during training, but how the code was laid out (minimize the expected value of deltas for changes from a superset of possible code changes), and comments (ratio of space vs mental effort to decode the repo for experienced vs inexperienced developers).
It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.
There is no hard takeoff
2 projects | news.ycombinator.com | 11 Aug 2023
I think this is a good casual introduction to the marketplace dynamics of how ML will impact the market. I do, however, disagree as this version of things assumes a more open-information set of competitive strategies among potentially ideal agents from a game theoretic perspective, and we can see this is absolutely not the case 'in real life'. To one of his examples -- Exxon-Mobil.
An updated version: There will be a log-normally distributed set of winners and losers from the exponential effects of ML and 'AI', and the flatness of this curve will be almost entirely solely determined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution.
While I'm here, I think Hotz does contribute a lot of good to the field, though I do have a bit of a minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George.
In Defense of Pure 16-Bit Floating-Point Neural Networks
2 projects | news.ycombinator.com | 23 May 2023
As a practitioner specializing in extremely fast-training neural networks, seeing a paper in 2023 considering fp32 as a gold standard over pure non-mixed fp16/bp16 is a bit shocking to me and feels dated/distracting from the discussion. They make good points but unless I am hopelessly misinformed, it's been pretty well established at this point in a number of circles that fp32 is overkill for the majority of uses for many modern-day practitioners. Loads of networks train directly in bfloat16 as the standard -- a lot of the modern LLMs among them. Mixed precision is very much no longer needed, not even with fp16 if you're willing to tolerate some range hacks. If you don't want the range hacks, just use bfloat16 directly. The complexity is not worth it, adds not much at all, and the dynamic loss scaler a lot of people use is just begging for more issues.
Both of the main repos that I've published in terms of speed benchmarks train directly in pure fp16 and bf16 respectively without any fp32 frippery, if you want to see an example of both paradigms successfully feel free to take a look (I'll note that bf16 is simpler on the whole for a few reasons, generally seamless): https://github.com/tysam-code/hlb-CIFAR10 [for fp16] and https://github.com/tysam-code/hlb-gpt [for bf16]
Personally from my experience, I think fp16/bf16 is honestly a bit too expressive for what we need, fp8 seems to do just fine and I think will be quite alright with some accommodations, just as with pure fp16. The what and the how of that is a story for a different day (and at this point, the max pooling operation is basically one of the slowest now).
You'll have to excuse my frustration a bit, it just is a bit jarring to see a streetsign from way in the past fly forward in the wind to hit you in the face before tumbling on its merry way. And additionally in the comment section the general discussion doesn't seem to talk about what seems to be a pretty clearly-established consensus in certain research circles. It's not really too much of a debate anymore, it works and we're off to bigger and better problems that I think we should talk about. I guess in one sense it does justify the paper's utility, but also a bit frustrating because it normalizes the conversation as a few notches back from where I personally feel that it actually is at the moment.
We've got to move out of the past, this fp32 business to me personally is like writing a Relu-activated VGG network in Keras on Tensorflow. Phew.
And while we're at it, if I shall throw my frumpy-grumpy hat right back into the ring, this is an information-theoretic problem! Not enough discussion of Shannon and co. Let's please fix that too. See my other rants for x-references to that, should you be so-inclined to punish yourself in that manner.
[P] 10x faster reinforcement learning HPO - now with CNNs!
3 projects | /r/MachineLearning | 5 Apr 2023
In a related but different vein (w/ hardcoded hyperparameters), if you'd like to have a research toolbench that trains rapidly on CIFAR10 (94% in <7 seconds on an A100), I made https://github.com/tysam-code/hlb-CIFAR10. It's also very breadboard-ized, for lack of a better term, so you can reclone and hack stuff in quickly to see if it works or doesn't. Most things I tested took 5 minutes or less, some a few seconds, and just a few more involved ones maybe half an hour to an hour or so, maybe a little more or less with debugging (depending upon how involved it was). I'm definitely curious about the software in this post though, as there was a lot of painful tuning involved (the reward space is, er, quite noisy).
MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention
2 projects | news.ycombinator.com | 2 Apr 2023
Karpathy's zero to hero series is excellent, and I really recommend it.
I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit.
[P] Introducing hlb-gpt: A rapid prototyping toolbench in <350 lines of code to speed up your LLM research exploration
2 projects | /r/MachineLearning | 5 Mar 2023
It's release day again and today we're releasing a new repository: hlb-gpt. It's based on nanoGPT, but smaller with an aggressively-trimmed feature set. In this initial release, the training performs almost exactly the same as Andrej's library, but a tiny bit faster and a tiny bit more accurate due to using PyTorch-native operators. We keep the complexity down by targeting tiny, rapid experiments on a single GPU only. The baseline network we're releasing gets <3.8 validation loss in just over 6 minutes. Having a rapidly training network offers a variety of benefits -- this is something that helped a lot when working on hlb-cifar10. Cycle times are king in research, and we rarely need giant models to get enough of a loss signal when prototyping/experimenting with a method.
[R] CIFAR10 in <8 seconds on an A100 (new architecture!)
2 projects | /r/MachineLearning | 12 Feb 2023
Reposting as the old post somehow pointed to an old release of mine! Strange! If you want to read the new release, you can do that here: https://github.com/tysam-code/hlb-CIFAR10/releases/tag/v0.4.02 projects | /r/MachineLearning | 12 Feb 2023
GPT in 60 Lines of NumPy
9 projects | news.ycombinator.com | 9 Feb 2023
That concept is not the easiest to describe succinctly inside a file like this, I think (especially as there are various levels of 'beginner' to take into account here). This is considered a very entry level concept, and I think there might be others who would consider it to be noise if logged in the code or described in the comments/blogpost.
After all, there was a disclaimer that you might have missed up front in the blogpost! "This post assumes familiarity with Python, NumPy, and some basic experience training neural networks." So it is in there! But in all of the firehose of info we get maybe it is not that hard to miss.
However, I'm here to help! Thankfully the concept is not too terribly difficult, I believe.
Effectively, the loss function compresses the task we've described with our labels from our training dataset into our neural network. This includes (ideally, at least), 'all' the information the neural network needs to perform that task well, according to the data we have, at least. If you'd like to know more about the specifics of this, I'd refer you to the original Shannon-Weaver paper on information theory -- Weaver's introduction to the topic is in plain English and accessible to (I believe) nearly anyone off of the street with enough time and energy to think through and parse some of the concepts. Very good stuff! An initial read-through should take no more than half an hour to an hour or so, and should change the way you think about the world if you've not been introduced to the topic before. You can read a scan of the book at a university hosted link here: https://raley.english.ucsb.edu/wp-content/Engl800/Shannon-We...
Using some of the concepts of Shannon's theory, we can see that anything that minimizes an information-theoretic loss function should indeed learn as well those prerequisites to the task at hand (features that identify xyz, features that move information about xyz from place A to B in the neural network, etc). In this case, even though it appears we do not have labels -- we certainly do! We are training on predicting the _next words_ in a sequence, and so thus by consequence humans have already created a very, _very_ richly labeled dataset for free! In this way, getting the data is much easier and the bar to entry for high performance for a neural network is very low -- especially if we want to pivot and 'fine-tune' to other tasks. This is because...to learn the task of predicting the next word, we have to learn tons of other sub-tasks inside of the neural network which overlap with the tasks that we want to perform. And because of the nature of spoken/written language -- to truly perform incredibly well, sometimes we have to learn all of these alternative tasks well enough that little-to-no-finetuning on human-labeled data for this 'secondary' task (for example, question answering) is required! Very cool stuff.
This is a very rough introduction, I have not condensed it as much as it could be and certainly, some of the words are more than they should be. But it's an internet comment so this is probably the most I should put into it for now. I hope this helps set you forward a bit on your journey of neural network explanation! :D :D <3 <3 :)))))))))) :fireworks:
For reference, I'm interested very much in what I refer to as Kolmogorov-minimal explanations (Wikipedia 'Kolmogorov complexity' once you chew through some of that paper if you're interested! I am still very much a student of it, but it is a fun explanation). In fact (though this repo performs several functions), I made https://github.com/tysam-code/hlb-CIFAR10 as beginner-friendly as possible. One does have to make some decisions to keep verbosity down, and I assume a very basic understand of what's happening in neural networks here too.
I have yet to find a good go-to explanation of neural networks as a conceptual intro (I started with Hinton -- love the man but extremely mathematically technical for foundation! D:). Karpathy might have a really good one, I think I saw a zero-to-hero course from him a little while back that seemed really good.
Andrej (practically) got me into deep learning via some of his earlier work, and I really love basically everything that I've seen the man put out. I skimmed the first video of his from this series and it seems pretty darn good, I trust his content. You should take a look! (Github and first video: https://github.com/karpathy/nn-zero-to-hero, https://youtu.be/VMj-3S1tku0)
For reference, he is the person that's made a lot of cool things recently, including his own minimal GPT (https://github.com/karpathy/minGPT), and the much smaller version of it (https://github.com/karpathy/nanoGPT). But of course, since we are in this blog post I would refer you to this 60 line numpy GPT first (A. to keep us on track, B. because I skimmed it and it seemed very helpful! I'd recommend taking a look at outside sources if you're feeling particularly voracious in expanding your knowledge here.)
I hope this helps give you a solid introduction to the basics of this concept, and/or for anyone else reading this, feel free to let me know if you have any technically (or-otherwise) appropriate questions here, many thanks and much love! <3 <3 <3 <3 :DDDDDDDD :)))))))) :)))) :))))
What are some alternatives?
hlb-gpt - Minimalistic, fast, and experimentation-friendly researcher's toolbench for GPT-like models in ~<365 lines of code. Reaches <3.8 validation loss on wikitext-103 on a single A100 in ~138 seconds.
label-errors - 🛠️ Corrected Test Sets for ImageNet, MNIST, CIFAR, Caltech-256, QuickDraw, IMDB, Amazon Reviews, 20News, and AudioSet
minGPT - A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training
nanoGPT - The simplest, fastest repository for training/finetuning medium-sized GPTs.
mnist_1_pt_2 - 1.2% test error on MNIST using only least squares and numpy calls.
SymbolicRegression.jl - Distributed High-Performance Symbolic Regression in Julia
tinygrad - You like pytorch? You like micrograd? You love tinygrad! ❤️
manim - Animation engine for explanatory math videos
umap_paper_notebooks - Notebooks in support of the UMAP paper
mono - monorepo for personal projects, experiments, ..
coalton - Coalton is an efficient, statically typed functional programming language that supercharges Common Lisp.