scraper
mwoffliner
Our great sponsors
| scraper | mwoffliner | |
|---|---|---|
| 1 | 7 | |
| 3 | 253 | |
| - | 1.6% | |
| 0.0 | 9.2 | |
| over 1 year ago | 8 days ago | |
| TypeScript | TypeScript | |
| Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
scraper
-
LinguaBook - React app for learning Basic English
Under the hood it is heavily using my Just-in-Time scraper (my another open source project). This scrapper parses HTML pages from multiple sources in the browser and the parsed results are displayed on the page.
mwoffliner
-
Wiktionary doesn’t support tables
You can also directly open a ticket at https://github.com/openzim/mwoffliner/issues with as much info as possible so we can look into it (zim name, language, date, article name, etc.)
-
Recent Wiktionary ZIM files don't show a search bar
Welp yes, that's a bug (likely a regression from a recent update). Can you please open a ticket at https://github.com/openzim/zim-requests/issues (we might move it later on but that's as good a starting place as can be).
-
Latest Wikipedia zim dump (97 GB) is available for download
https://github.com/openzim/mwoffliner/issues/1655 Unfortunate that there's no way to convert the easiest way to make proper dumps of wikis (ArchiveTeam's wikiteam-tools) to Kiwix Zims. That would allow for all sorts of niche information to be preserved in a readable way.
-
What's the "best" way to make your own ZIMs (in docker)?
I'm looking at making my own ZIM though not sure the best way to go about it. I've seen zimit on Github and the mwoffliner on Github too.
-
Self made ZIM-File only contains [object object]
Generally speaking, I'd advise opening a ticket on https://github.com/openzim/mwoffliner/issues
-
Creating ZIM files for Kiwix by myself?
r/kiwix would be the place to ask, but at the end of the day it all comes down to heading out to openzim.org (or the corresponding github repo) and figuring it out. You can either grab zimit and run it locally, or access all the libraries that will help you build your own scraper (Nautilus will assemble documents and videos into a single file library, MWoffliner will do for wikis, youtube will do YouTube, etc.).
What are some alternatives?
extension - web scraping extension
wikipedia-mirror - 🌐 Guide and tools to run a full offline mirror of Wikipedia.org with three different approaches: Nginx caching proxy, Kiwix + ZIM dump, and MediaWiki/XOWA + XML dump
cheerio - The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
wikiscript - wikiscript gem - scripts for wikipedia (get wikitext for page, parse tables & links, etc.)
scraper - Nodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless clients: Puppeteer, Playwright, Cheerio, JSdom.
nautilus - Turns a collection of documents into a browsable ZIM file
linguabook.github.io - Just-in-time scraper of linguistic information from different sources like Cambridge, Merriam-Webster. It also has a satellite Chrome Extension to see linguistic information for selected word while you surfing the internet
zimit - Make a ZIM file from any Web site and surf offline!
cheerio - Fast, flexible, and lean implementation of core jQuery designed specifically for the server. [Moved to: https://github.com/cheeriojs/cheerio]
kiwix-tools - Command line Kiwix tools: kiwix-serve, kiwix-manage, ...
libkiwix - Common code base for all Kiwix ports