-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
We believe in fostering open access to the best models, datasets, and hardware. So we have made the model, training recipe, weights, and checkpoints available on Hugging Face and GitHub under the permissive Apache 2.0 license. Our paper, which will be available soon, will detail our training methods and performance results. Please see figure 1 for a summary of how the Cerebras-GPT family compares to industry-leading models.
This is the first time I have seen muP applied by the third party. See Cerebras Model Zoo, where muP models have scale-invariant constant LR.
Related posts
-
Bard is getting better at logic and reasoning
-
OpenAI’s policies hinder reproducible research on language models
-
[R] Greg Yang's work on a rigorous mathematical theory for neural networks
-
DeepMind’s New Language Model,Chinchilla(70B Parameters),Which Outperforms GPT-3
-
"Training Compute-Optimal Large Language Models", Hoffmann et al 2022 {DeepMind} (current LLMs are significantly undertrained)