bookcorpus
instagram-scraper
| bookcorpus | instagram-scraper | |
|---|---|---|
| 3 | 2 | |
| 778 | 2,503 | |
| - | - | |
| 3.1 | 3.3 | |
| 10 months ago | almost 2 years ago | |
| Python | Python | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
bookcorpus
- Show HN: New AI Dataset Based on LibGen and Sci-Hub
- Can chat GPT overtake Google if they play their cards right?
-
On the Danger of Stochastic Parrots [pdf]
The GPT-3 paper (section 2.2) mentions using two datasets referred to as "books1" and "books2", which are 12B and 55B byte pair encoded tokens each.
Project Gutenberg has 3B word tokens I believe, so it seems like it could be one of them, assuming the ratio of word tokens to byte-pair tokens is something like 3:12 to 3:55.
Another likely candidate alongside Gutenberg is libgen, apparently, and looks like there have been successful efforts to create a similar dataset called bookcorpus: https://github.com/soskek/bookcorpus/issues/27). The discussion on that github issue suggests bookcorpus is very similar to "books2", which would make gutenberg "books1"?
This might be why the paper is intentionally vague about the books used?
instagram-scraper
-
Help in scraping Instagram
Yes: https://github.com/realsirjoe/instagram-scraper
-
Fetching all comments from an Instagram post?
You could look into this Python scraper, but it's a bit buggy. There's also this PHP scraper you can use to get all the comments.
What are some alternatives?
awesome-chatgpt-prompts - This repo includes ChatGPT prompt curation to use ChatGPT better.
instagram-php-scraper - Get account information, photos, videos, stories and comments.
Replicate-Toronto-BookCorpus - This repository contains code to replicate the no-longer publicly available Toronto BookCorpus dataset
GramAddict bot - Completely free and open-source human-like Instagram bot. Powered by UIAutomator2 and compatible with basically any Android device 5.0+ that can run Instagram - real or emulated.
trafilatura - Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
cloudflare-scrape - A Python module to bypass Cloudflare's anti-bot page.
open-discourse - Open Discourse is the first fully comprehensive corpus of the plenary proceedings of the federal German Parliament (Bundestag).
instascrape - Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically
stc - Distributed free search engine and AI tools that grant access to knowledge
fansly-downloader - Easy to use fansly.com content downloading tool. Written in python, but ships as a standalone Executable App for Windows too. Enjoy your Fansly content offline anytime, anywhere in the highest possible content resolution! Fully customizable to download in bulk or single: photos, videos & audio from timeline, messages, collection & specific posts 👍
korean-word-ipa-dictionary - Dictionary of pairs of Korean word and IPA crawled from Wiktionary (Korean edition)
Moodle-DL - Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)