docarray
img2dataset
Our great sponsors
docarray | img2dataset | |
---|---|---|
32 | 13 | |
2,730 | 3,225 | |
2.1% | - | |
9.2 | 7.3 | |
6 days ago | 2 days ago | |
Python | Python | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
docarray
-
I plan to build my own AI powered search engine for my portfolio. Do you know ones that are open-source?
For some alternatives, I know there’s DocArray where you can handle text, image and audio data. is basically a toolbox for multimodal data and then there should be Haystack which is also let you build search systems and also has to do something with Transformers and LLMs.
-
A Guide to Using OpenTelemetry in Jina for Monitoring and Tracing Applications
DocArray to manipulate data and interact with the storage backend using document store.
-
This week(s) in DocArray
It's already been two weeks since the last alpha release of DocArray v2. And since then a lot has happened — we've merged features we're really proud of, and we've cried tears of joy and misery trying to coerce Python into doing what we want. If you want to learn about interesting Python edge cases or follow the advancement of DocArray v2 development then you’ve come to the right place in this blog post!
-
Improving Search Quality for Non-English Queries with Fine-tuned Multilingual CLIP Models
The German Fashion12k dataset is available for free use by the Jina AI community. After logging into Jina AI Cloud, you can download it directly in DocArray format:
The German Fashion12k dataset is available for free use by the Jina AI community. After logging into Jina AI Cloud, you can download it directly in DocArrayformat:
-
Looking for open source projects in Machine Learning and Data Science
You could try spaCy. This is the brains of the operation - an open-source NLP library for advanced NLP in Python. Another is DocArray - It's built on top of NumPy and Dask, and good for preprocessing, modeling, and analysis of text data.
-
What are the best Python libraries to learn for beginners?
DocArray: Multimodal Data Library
-
How to Use Every Vector Database in Python with DocArray
As the description suggests on the project home page, DocArray is a library for nested, unstructured and multimodal data.
-
Announcing DocArray v2
Hey reddit, we are Johannes and Sami from the DocArray team and we are super excited to announce version 2 of the DocArray library!
-
Most Trending Open Source MLOps Tools of 2022
Document is the fundamental data structure. (This project is also an opensource project by the Linux Foundation)
img2dataset
-
Who kept the bots out? Stopping content being harvested by AI
The particular tool mentioned in the Vice article is Img2dataset, and right now, it doesn't pay attention to the robots.txt file, the normal mechanism you can use to dissuade well behaved bots from indexing your content. However, it does respect a new HTTP header directive, X-Robots-Tag: noai (and also noindex, though that's an existing and already well-known part of the robots.txt standard).
-
An AI Scraping Tool Is Overwhelming Websites with Traffic
The established norm is that scrapers have to download robots.txt and support the standard robots.txt features, notably including `Crawl-Delay` which sets a rate limit. This is the established standard by which websites tell scrapers what the rules are for scraping them.
This tool is scraping sites, it has webmasters reporting actual disruption, it doesn't have robots.txt support. When people complained (eg in https://github.com/rom1504/img2dataset/issues/48), the author's stance was basically "PRs welcome". It looks like a third party recently contributed a PR to make it respect robots.txt (https://github.com/rom1504/img2dataset/pull/302), albeit without `Crawl-Delay` support, which is not merged yet.
I have seen the same thing with other recent AI tools (eg https://github.com/m1guelpf/browser-agent/issues/2) and I think it's important to defend the robots.txt convention and nip this in the bud. If a bot doesn't make a reasonable effort to respect robots.txt and it causes disruption, it's a denial-of-service attack and should be treated as such. No excuses.
-
Please make this tool “opt-in” by default
//First sentence unchanged
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
//Then pull up the policy link first
To understand why image creators and artists may choose to specify such headers for their images, and choose to actively not consent to their images being collected, see [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact).
AI training would not be possible without the contribution of artists, and it is the recommendation of this tool's authors that you should respect their communicated wishes. However, if you have a legitimate reason to bypass these headers, to disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
```
Opt-out directives
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
See [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact) to understand better why you may decide to enable or disable this feature.
-
Stable Attribution
it's not remembering pixels. for it to do that, it would have to have the pixels stored somewhere. It does not.
The laion 5b dataset is in the neighborhood of 220TB. (1) That is how much storage space you need to remember the pixels.
The stable diffusion 1.5 checkpoint is 7gb. (2)
1 https://github.com/rom1504/img2dataset/blob/main/dataset_exa...
2 https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/m...
-
Using AI generated art to show the history of the world
Well, not all of it. LAION-5B filtered out illegal content, but not questionable content — the latter was just rated. LAION-5B provides punsafe, pwatermark and aesthetic scores. laion-aesthetic (https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-aesthetic.md) takes all of them into account.
What are some alternatives?
Milvus - A cloud-native vector database, storage for next generation AI applications
transformers - 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
bootcamp - Dealing with all unstructured data, such as reverse image search, audio search, molecular search, video analysis, question and answer systems, NLP, etc.
chitra - A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.
kaggle-environments
imodels - Interpretable ML package 🔍 for concise, transparent, and accurate predictive modeling (sklearn-compatible).
discoart - 🪩 Create Disco Diffusion artworks in one line
habitat-sim - A flexible, high-performance 3D simulator for Embodied AI research.
google-images-download - Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
nn - 🧑🏫 60 Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠
habitat-lab - A modular high-level library to train embodied AI agents across a variety of tasks and environments.
dalle-mini - DALL·E Mini - Generate images from a text prompt