SmallPebble VS GPU-Puzzles

You can implement autograd as a library. Just take a look at this

https://github.com/sradc/SmallPebble

The first line of the description is:

  > SmallPebble is a minimal automatic differentiation and deep learning library written from scratch in Python, using NumPy/CuPy.

Thanks for this. My approach to speeding up an autodiff system like this was to write it in terms of nd-arrays rather than scalars, using numpy/cupy [1]. But it's still slower than deep learning frameworks that compile / fuse operations. Wondering how it compares to the approach in this post. (Might try to benchmark at some point.)

[1] https://github.com/sradc/SmallPebble

Located here: https://github.com/sradc/SmallPebble

I found it helpful to start with CUDA on numba since it lets you write GPU kernels in python. Assuming you're like most ML engineers and you're more familiar with python than C++, this allows you to separately learn CUDA concepts from also learning C++ at the same time. There's also a set of GPU puzzles for beginners [1] using to get started with numba CUDA.

[1] https://github.com/srush/GPU-Puzzles

Are you having fun with Machine Learning? Go and teach yourself beginner GPU programming with this wonderful notebook: GitHub repo

Practice puzzles: https://github.com/srush/GPU-Puzzles