Github-Ranking VS transformers

5th highest number of stars of any repo on GitHub 🙃

This explains the uneven distribution of Haskell applications, but this does not explain why the distribution is more even in other languages. But is that even the case? You mention Python, and Python happens to be THE language of choice for data science projects, so I would expect to also see an uneven distribution there. And Java happens to be THE language of choice for writing Android applications, so I would expect an uneven distribution there too. And Rust is a systems programming language, so I would expect games and other things that really need to run fast. Let's look at lists of popular projects by language:

The repo I chose was Github-Ranking, a repo to check the most starred and forked GitHub repos of the day. The link can be found here: https://github.com/EvanLi/Github-Ranking. I picked this repo because I've never explored the most popular repos before and this allowed me to see what a lot of people are working on.

Clocking in at 190K Github stars React's github ranking is easily ranked in the top 10.

Here's a few other tools that are written in Perl, sorted by GitHub popularity: https://github.com/EvanLi/Github-Ranking/blob/master/Top100/...

Actually, that repo has lists like this for most languages: https://github.com/EvanLi/Github-Ranking

Is t5x an encoder/decoder architecture?

Some more general options.

The Flax ecosystem

https://github.com/google/flax?tab=readme-ov-file

or dm-haiku

https://github.com/google-deepmind/dm-haiku

were some of the best developed communities in the Jax AI field

Perhaps the “trax” repo? https://github.com/google/trax

Some HF examples https://github.com/huggingface/transformers/tree/main/exampl...

Sadly it seems much of the work is proprietary these days, but one example could be Grok-1, if you customize the details. https://github.com/xai-org/grok-1/blob/main/run.py

The HuggingFace transformers library already has support for a similar method called prompt lookup decoding that uses the existing context to generate an ngram model: https://github.com/huggingface/transformers/issues/27722

I don't think it would be that hard to switch it out for a pretrained ngram model.

Most of this tutorial is based on Hugging Face course about Transformers and on Niels Rogge's Transformers tutorials: make sure to check their work and give them a star on GitHub, if you please ❤️

* Superconvergence + LR range finder + Fast AI's Ranger21 optimizer was the goto optimizer for CNNs, and worked fabulously well, but on transformers, the learning rate range finder sadi 1e-3 was the best, whilst 1e-5 was better. However, the 1 cycle learning rate stuck. https://github.com/huggingface/transformers/issues/16013

Thanks! :) I'm pushing them into transformers, pytorch-gemma and collabing with the Gemma team to resolve all the issues :)

The RoPE fix should already be in transformers 4.38.2: https://github.com/huggingface/transformers/pull/29285

My main PR for transformers which fixes most of the issues (some still left): https://github.com/huggingface/transformers/pull/29402

If you want to tinker with the architecture Hugging Face has a FOSS implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/tr...

If you want to reproduce the training pipeline, you couldn't do that even if you wanted to because you don't have access to thousands of A100s.

I am aware that using mixed precision reduces the stability of weight and there will be little consistency but don't expect it to be this much. I have attached the graph of evaluation metrics. If someone can give me some insight into this issue, that would be great.