snowflake-arctic reviews and mentions

• Snowflake Arctic Instruct (128x3B Moe LLM)
  2 projects | news.ycombinator.com | 24 Apr 2024

  By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing:

  i) Open-Sourced Model Weights

  ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/

  iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.

  iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-serie...

  v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.