reweight-gpt VS numerical-linear-algebra

Instead of self attention, I tried to generate the self-attention matrix directly using lateral connections among the inputs. The method is like LSTM but it gates all the past inputs using separate gates for each input (it can be parallelized). It's very easy to implement the method into the current GPT architectures. You just remove the attention part and replace it with learnable weights. Her is a working implementation (around100 lines!): Code: https://github.com/hunar4321/reweight-gpt In my experience, it learns very well and it can super-pass the self-attention mechanism if the number of the parameters are matched. (I tested it on small datasets for next character prediction. I haven't systematically compared these two methods yet).

If you really like to dig into math, I liked the Udacity course on Intro to Deeplearning with Pytorch. Also, the Stanford course CS231n Convolutional Neural Networks for Visual Recognition is a good place to understand some basics. Other two courses to get you jumpstarted are Practical Deep Learning for Coders and Linear Algebra Course by FastAI

I've also bookmarked Fast.Ai Computational Linear Algebra for Coders. https://github.com/fastai/numerical-linear-algebra/blob/master/README.md

In that case, I would probably be temped to teach a numerical methods of linear algebra course using NumPy / Numba. Something like https://github.com/fastai/numerical-linear-algebra or https://pythonnumericalmethods.berkeley.edu/notebooks/Index.html

Computation Linear Algebra Lectures Study Material To do after completing curricula.