reweight-gpt VS ML-foundations

Instead of self attention, I tried to generate the self-attention matrix directly using lateral connections among the inputs. The method is like LSTM but it gates all the past inputs using separate gates for each input (it can be parallelized). It's very easy to implement the method into the current GPT architectures. You just remove the attention part and replace it with learnable weights. Her is a working implementation (around100 lines!): Code: https://github.com/hunar4321/reweight-gpt In my experience, it learns very well and it can super-pass the self-attention mechanism if the number of the parameters are matched. (I tested it on small datasets for next character prediction. I haven't systematically compared these two methods yet).

As others have said, you won't need calculus immediately, but it's important that you make a good attempt at learning up to Calc3. I also didn't have a math heavy undergrad so it took a lot of self-study for me, but it's possible. Simulation has a great math boot camp at the beginning to review everything but you'll want to be prepped with Calc before that because that class is all calculus based probability. Some other good resources are the 3Blue1Brown videos on YouTube. They have a great series for both calc & linear algebra to talk through all the intuition with visuals. I also really like John Krohns series because you code through the math which is very applicable for us in this program. I only did his linear Algebra, but he has a whole series with Calc and probability, too. https://github.com/jonkrohn/ML-foundations