DALLE-mtf
google-research
DALLE-mtf | google-research | |
---|---|---|
41 | 98 | |
435 | 32,863 | |
0.0% | 0.7% | |
0.0 | 9.6 | |
about 2 years ago | about 3 hours ago | |
Python | Jupyter Notebook | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DALLE-mtf
-
How Open is Generative AI? Part 2
This vision is in line with EleutherAI, a non-profit organization founded in July 2020 by a group of researchers. Driven by the perceived opacity and the challenge of reproducibility in AI, their goal was to create leading open-source language models.
- The open source learning curve for AI researchers
- EleutherAI: Empowering Open-Source Artificial Intelligence Research
-
Seeking advice on fine-tuning Pythia for semantic search in a non-English language
My current idea is to utilize the EleutherAI pythia (Databricks Dolly). I would like to know whether translating the Dolly-15k dataset into the desired language using state-of-the-art translation techniques like DeepL would be a viable approach to fine-tune the Pythia base model. I want to use this model for semantic search, so perfection is not a necessity.
-
Does anyone want to collaborate to make anti-capitalist AI?
There are open source AI efforts, like EleutherAI. Needless to say, they are lagging behind big players, but it's better than nothing.
-
ChatGPT is bonkers.
The new GPT 3.5 isn't aware what are GPT-3.5 or davinci-002 (repeatable) and claimed that it was designed by EleutherAI and has only 6 bil parameters (wasn't been able to repeat but didn't really try).
-
My teacher has falsely accused me of using ChatGPT to use an assignment.
Hi, my name is Stella Biderman and I run EleutherAI, the one of the foremost non-profit research institutes in the world that trains and studies large language models. I have been involved with the majority of models to hold the title “largest open source GPT model in the world” and have dabbled in exploring using plagiarism detection tools to identify code written by GPT-J.
-
dolly-v2-12b
dolly-v2-12bis a 12 billion parameter causal language model created by Databricks that is derived from EleutherAI’s Pythia-12b and fine-tuned on a ~15K record instruction corpus generated by Databricks employees and released under a permissive license (CC-BY-SA)
-
Futurism: "The Company Behind Stable Diffusion Appears to Be At Risk of Going Under"
It is true that Emad needs to find an appropriate business model. The good news is that the hype is still undergoing. I'm sure that Emad can grab another round of liquidity injection. He got plenty of resources. Remember he is also from the finance industry. He got https://www.eleuther.ai/ which can supply a secured, in-house custom LLM equivalent to bloombergGPT.
-
How can AI be used to protect against exploitative use of other AI?
By promoting fully open-source AI, i.e. making datasets, models, methodology and codebases freely available and transparent. What OpenAI claimed to be aiming for, basically.
google-research
-
Show HN: Next-token prediction in JavaScript – build fast LLMs from scratch
People on here will be happy to say that I do a similar thing, however my sequence length is dynamic because I also use a 2nd data structure - I'll use pretentious academic speak: I use a simple bigram LM (2-gram) for single next-word likeliness and separately a trie that models all words and phrases (so, n-gram). Not sure how many total nodes because sentence lengths vary in training data, but there are about 200,000 entry points (keys) so probably about 2-10 million total nodes in the default setup.
"Constructing 7-gram LM": They likely started with bigrams (what I use) which only tells you the next word based on 1 word given, and thought to increase accuracy by modeling out more words in a sequence, and eventually let the user (developer) pass in any amount they want to model (https://github.com/google-research/google-research/blob/5c87...). I thought of this too at first, but I actually got more accuracy (and speed) out of just keeping them as bigrams and making a totally separate structure that models out an n-gram of all phrases (e.g. could be a 24-token long sequence or 100+ tokens etc. I model it all) and if that phrase is found, then I just get the bigram assumption of the last token of the phrase. This works better when the training data is more diverse (for a very generic model), but theirs would probably outperform mine on accuracy when the training data has a lot of nearly identical sentences that only change wildly toward the end - I don't find this pattern in typical data though, maybe for certain coding and other tasks there are those patterns though. But because it's not dynamic and they make you provide that number, even a low number (any phrase longer than 2 words) - theirs will always have to do more lookup work than with simple bigrams and they're also limited by that fixed number as far as accuracy. I wonder how scalable that is - if I need to train on occasional ~100-word long sentences but also (and mostly) just ~3-word long sentences, I guess I set this to 100 and have a mostly "undefined" trie.
I also thought of the name "LMJS", theirs is "jslm" :) but I went with simply "next-token-prediction" because that's what it ultimately does as a library. I don't know what theirs is really designed for other than proving a concept. Most of their code files are actually comments and hypothetical scenarios.
I recently added a browser example showing simple autocomplete using my library: https://github.com/bennyschmidt/next-token-prediction/tree/m... (video)
And next I'm implementing 8-dimensional embeddings that are converted to normalized vectors between 0-1 to see if doing math on them does anything useful beyond similarity, right now they look like this:
[nextFrequency, prevalence, specificity, length, firstLetter, lastLetter, firstVowel, lastVowel]
- Google Research website is down
-
Jpegli: A New JPEG Coding Library
The change was literally just made: https://github.com/google-research/google-research/commit/4a...
It appears this was in response to Hacker News comments.
- Multi-bitrate JPEG compression perceptual evaluation dataset 2023
-
Vector Databases: A Technical Primer [pdf]
There are options such as Google's ScaNN that may let you go farther before needing to consider specialized databases.
https://github.com/google-research/google-research/blob/mast...
-
Labs.Google
I feel it was unnecesary to create this because https://research.google/ already exists? It just seems like they want to take another URL with a "pure" domain name instead of psubdirectories, etc parts.
-
Smerf: Streamable Memory Efficient Radiance Fields
https://github.com/google-research/google-research/blob/mast...
-
Shisa 7B: a new JA/EN bilingual model based on Mistral 7B
You could also try some dedicated translation models like https://huggingface.co/facebook/nllb-moe-54b (or https://github.com/google-research/google-research/tree/master/madlad_400 for something smaller) and see how they do.
-
Translate to and from 400+ languages locally with MADLAD-400
Google released T5X checkpoints for MADLAD-400 a couple of months ago, but nobody could figure out how to run them. Turns out the vocabulary was wrong, but they uploaded the correct one last week.
- Mastering ROUGE Matrix: Your Guide to Large Language Model Evaluation for Summarization with Examples
What are some alternatives?
VQGAN-CLIP - Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.
qdrant - Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/
CLIP-Guided-Diffusion - Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.
fast-soft-sort - Fast Differentiable Sorting and Ranking
dalle-mini - DALL·E Mini - Generate images from a text prompt
faiss - A library for efficient similarity search and clustering of dense vectors.
big-sleep - A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN. Technique was originally created by https://twitter.com/advadnoun
ml-agents - The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
gpt-3 - GPT-3: Language Models are Few-Shot Learners
Milvus - A cloud-native vector database, storage for next generation AI applications
DALLE-pytorch - Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
struct2depth - Models and examples built with TensorFlow