habitat-sim VS transformers

Spot on! I think you grasped the general idea. During some of my 3d studies, I collected data about hand movements, eye movements and navigation paths within scenes, which could potentially be used for training data in robots (e.g. to train robot arm-suction grip, visual input and navigation respectively). I see projects like this https://aihabitat.org/, where my research seems quite relevant.

I need an easy to set up simulation of a 3d environment and I was wondering what you guys are using. Something like https://aihabitat.org/ . It already comes with rich visuals, which is quite important in my case and it works out of the box so I don't need to waste time developing my own models and graphics. Unfortunately habitat ai doesn't work on windows. Are there some alternatives?

There are plenty of them out there. I spend a lot of time contributing to open source projects like Habitat-Sim https://github.com/facebookresearch/habitat-sim and Habitat-Lab https://github.com/facebookresearch/habitat-lab which have a ton of open issues and code maintaince stuff that we would welcome contributions of.

With some hacks it is pretty easy to get drones working in Habitat-Sim: https://github.com/facebookresearch/habitat-sim

The HuggingFace transformers library already has support for a similar method called prompt lookup decoding that uses the existing context to generate an ngram model: https://github.com/huggingface/transformers/issues/27722

I don't think it would be that hard to switch it out for a pretrained ngram model.

Most of this tutorial is based on Hugging Face course about Transformers and on Niels Rogge's Transformers tutorials: make sure to check their work and give them a star on GitHub, if you please ❤️

* Superconvergence + LR range finder + Fast AI's Ranger21 optimizer was the goto optimizer for CNNs, and worked fabulously well, but on transformers, the learning rate range finder sadi 1e-3 was the best, whilst 1e-5 was better. However, the 1 cycle learning rate stuck. https://github.com/huggingface/transformers/issues/16013

Thanks! :) I'm pushing them into transformers, pytorch-gemma and collabing with the Gemma team to resolve all the issues :)

The RoPE fix should already be in transformers 4.38.2: https://github.com/huggingface/transformers/pull/29285

My main PR for transformers which fixes most of the issues (some still left): https://github.com/huggingface/transformers/pull/29402

If you want to tinker with the architecture Hugging Face has a FOSS implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/tr...

If you want to reproduce the training pipeline, you couldn't do that even if you wanted to because you don't have access to thousands of A100s.

I am aware that using mixed precision reduces the stability of weight and there will be little consistency but don't expect it to be this much. I have attached the graph of evaluation metrics. If someone can give me some insight into this issue, that would be great.

In transformers, they tried really hard to have a single function or method to deal with both self and cross attention mechanisms, masking, positional and relative encodings, interpolation etc. While it allows a user to use the same function/method for any model, it has led to severe parameter bloat. Just compare the original implementation of llama by FAIR with the implementation by HF to get an idea.