metaseq VS manim

This is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch for LLMs outside of Google.) I hope this newco eventually releases a more technical report about their training adventures, like the PDF file here: https://github.com/facebookresearch/metaseq/tree/main/projec...

The number of people who can actually pre-train a true LLM is very small.

It remains a major feat with many tweaks and tricks. Case in point: the 114 pages of OPT175B logbook [1]

[1] https://github.com/facebookresearch/metaseq/blob/main/projec...

Reliability at scale: take a look at the OPT training log book for their 175B model run. It needed a lot of babysitting. In my experience, that scale of TPU training run requires a restart about once every 1-2 weeks—and they provide the middleware to monitor the health of the cluster and pick up on hardware failures.

I really appreciated this log of Facebook training a large language model of how troublesome AI development can be: https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles

Stable Diffusion will run on any decent gaming GPU or a modern MacBook, meanwhile LLMs comparable to GPT-3/ChatGPT have had pretty insane memory requirements - e.g., <https://github.com/facebookresearch/metaseq/issues/146>

It seems so, check this log book from Meta: https://github.com/facebookresearch/metaseq/blob/main/projec...

That is definitely one of the things he does better than most. He actually wrote a custom library for math animations: https://github.com/3b1b/manim

Not quite what you're looking for, but worth pointing out that Grant Sanderson of 3Blue1Brown has published the "framework" he uses for his math videos on GitHub.

https://github.com/3b1b/manim

3b1b uses a python library for creating those videos.

https://github.com/3b1b/manim

Manim, 3b1b's animation library is open source: https://github.com/3b1b/manim

Coming to motion design, this rumour takes of due to the fact that there are programming libraries like Manim and Motion-Canvas which are actually used to generate animations from code. You can search 3Blue1Brown channel on youtube.

It kinda looks like this: https://github.com/3b1b/manim, but that would be a crazy usage of it. Wondering if they’re compositing Manim with a more traditional animation suite.

Probably used this for the animation: https://github.com/3b1b/manim

3blue1brown has had success with (their tool) manim, which uses Python.