mation-spec VS trax

Hi, I've created an application where you can follow step by step origami fold instructions, and a Creator where you can make these interactive folds.

On comparing to video instructions, you have the ability to quickly skip/rewind steps and replay a complicated step many times.

On the creation side, there have been one or two attempts at this before, but those solutions rely on mouse drags as the user interface. This greatly limited the kinds of folds possible. The foldMation Creator uses commands, keywords and values to compose a domain specific language/step and provides a (relatively speaking) easy to use user interface to compose the steps.

For those interested in using the Creator, please go through the tutorial at the top of the create page.

Btw, the DSL for foldMation uses https://github.com/mationai/mation-spec. I created it since I couldn't find anything out there that is similar, allowing me to specify a well structured data with English-like readable syntax.

Let me know what you think?

The DSL for foldMation uses https://github.com/mationai/mation-spec . I created it since I couldn't find anything out there that is similar, allowing me to specify a well structured data with English-like readable syntax.

Ohm is a wonderful tool. I used it to create mation-spec [0], a readable structured configuration and specification format to automate and run code. I look hard trying to find something like it before giving up and creating one myself with the help of Ohm. The mation-spec is the basis of an origami fold simulation language to describe and simulate origami folds. PM me if you like to see it before I post the simulator on HN.

[0] https://github.com/mationai/mation-spec

Have you thought of finding or creating something like this [0]?

I created this as the basis for my origami folding descriptive language. I tried to find something similar, requirements being both well structured and English-like but couldn't find any, so I created it.

The origami folding app will hopefully be out in 2 weeks, so you can see how it's used.

[0] https://github.com/fuzzthink/mation-spec

Is t5x an encoder/decoder architecture?

Some more general options.

The Flax ecosystem

https://github.com/google/flax?tab=readme-ov-file

or dm-haiku

https://github.com/google-deepmind/dm-haiku

were some of the best developed communities in the Jax AI field

Perhaps the “trax” repo? https://github.com/google/trax

Some HF examples https://github.com/huggingface/transformers/tree/main/exampl...

Sadly it seems much of the work is proprietary these days, but one example could be Grok-1, if you customize the details. https://github.com/xai-org/grok-1/blob/main/run.py

and the implementation https://github.com/google/trax/blob/master/trax/models/resea... if you are interested.

Hope you get to look into this!

Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset!

I'm not sure if this is the right place to ask about this, but could you consider training an LLM using a more advanced, sparse transformer architecture (specifically, "Terraformer" from this paper https://arxiv.org/abs/2111.12763 and this codebase https://github.com/google/trax/blob/master/trax/models/resea... by Google Brain and OpenAI)? I understand the pressure to focus on training a straightforward LLaMA replication, but of course you see that it's a legacy dense architecture which limits its inference performance. This new architecture is not just an academic curiosity but is already validated at scale by Google, providing 10x+ inference performance boost on the same hardware.

Frankly, the community's compute budget - for training and for inference - isn't infinite, and neither is the public's interest in models that do not have advantage (at least in convenience) over closed-source ones; and so we should utilize both those resources as efficiently as possible. It could be a big step forward if you trained at least LLaMA-Terraformer-7B and 13B foundation models on the whole dataset.

But a couple years later they came out with open source implementations yeah: https://github.com/google/trax/tree/master/trax/models/reformer

Code: https://github.com/google/trax/blob/master/trax/examples/Terraformer_from_scratch.ipynb

Training language models is expensive. Train the biggest model you can afford. I assume you've tried the colab from the reformer GitHub: https://github.com/google/trax/tree/master/trax/models/reformer