Tinygrad: A simple and powerful neural network framework

• tinygrad

  58 17,800 9.7 Python

  Discontinued You like pytorch? You like micrograd? You love tinygrad! ❤️ [Moved to: https://github.com/tinygrad/tinygrad] (by geohot)

  

> - MovementOps operate on one tensor and move the data around, copy-free with ShapeTracker. RESHAPE, PERMUTE, EXPAND, etc...

> But how...where are your CONVs and MATMULs? Read the code to solve this mystery.

Ok, I was curious, so I read the code. The answer it represents a MATMUL as a 1x1 CONV. And it lied about CONV, which is a ProcessingOps.CONV and explicitly represented and implemented: https://github.com/geohot/tinygrad/blob/c0050fab8ff0bc667e40...

• thinc

  4 2,796 7.6 Python

  🔮 A refreshing functional take on deep learning, compatible with your favorite libraries

  

I love those tiny DNN frameworks, some examples that I studied in the past (I still use PyTorch for work related projects) :

thinc.by the creators of spaCy https://github.com/explosion/thinc

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• nnabla

  1 2,694 8.2 Python

  Neural Network Libraries

  

• black

  322 37,456 9.4 Python

  The uncompromising Python code formatter

  

It's funny that geohot/tinygrad chooses to not meet the PEP8 standards [0] just to stay on brand (<1000 lines). Black [1] or any other python autoformatter would probably 2x the lines of code.

[0] https://peps.python.org/pep-0008/

[1] https://github.com/psf/black

• jittor

  4 2,998 7.6 Python

  Jittor is a high-performance deep learning framework based on JIT compiling and meta-operators.

Very similar idea as Jittor, convolution definitely can be break down: https://github.com/Jittor/jittor/blob/master/python/jittor/n...

• loop_tool

  4 145 0.0 C++

  Discontinued A thin, highly portable toolkit for efficiently compiling dense loop-based computation.

  

I've done some work in the past in representations and you actually can represent Conv and MatMul in more primitive ways. I ended up writing an IR called loop_tool that exposes this stuff pretty nicely:

https://github.com/facebookresearch/loop_tool/blob/main/pyth...

The idea is basically this: https://news.ycombinator.com/item?id=28883086

• shumai

  15 1,122 2.2 TypeScript

  Fast Differentiable Tensor Library in JavaScript and TypeScript with Bun + Flashlight

  

Doesn’t really matter for large batch/large model training on GPUs that don’t need much coordination.

But Python speed is one of the main motivations for a JS/TS based ML lib I’m working on: https://github.com/facebookresearch/shumai

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• openpilot

  839 47,873 10.0 Python

  openpilot is an open source driver assistance system. openpilot performs the functions of Automated Lane Centering and Adaptive Cruise Control for 250+ supported car makes and models.

  

As it was recently discussed at length here on HN [0] (401 comments), George Hotz (the lead of tinygrad) is taking time off his self-driving startup comma.ai [1]. Curious if this would help or hurt tinygrad progress.

[0] https://news.ycombinator.com/item?id=33406790

[1] https://comma.ai/

• Pytorch

  340 78,205 10.0 Python

  Tensors and Dynamic neural networks in Python with strong GPU acceleration

  

Thanks, that's certainly helpful anecdotal evidence.. yeah it seems like there should be an "AllNorm" implementation that covers all cases and is just fast. I was wondering because I'm currently looking at math_group_norm, which was ported from PyTorch/XLA and it results in a really weird decomposition that I'm astonished works at all. https://github.com/pytorch/pytorch/blob/master/aten/src/ATen...

I'm also wondering if the handcoded backward passes are actually "numerically correct", because e.g. epsilon doesn't appear in it at all. Someone worked out the gradients manually for BN here: https://web.archive.org/web/20180826123459/http://cthorey.gi...

You can clearly see epsilon appearing in the output. And of course there's the whole training vs. eval mode thing with BN which GN doesn't have.

In any case, thanks again.

Suggest a related project