grab-site VS Joplin

The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site (https://wiki.archiveteam.org/index.php/Who_We_Are) but I found that in less than 15 seconds of searching so do your own diligence.

You can use grab-site with --no-offsite-links and --igsets=mediawiki.

Does the forum provide real-time notification for new posts? Like maybe a RSS feed, or a 'New' section? If so, some scripting around grab-site or httrack could grab them quickly.

After a lot of searching for a similar topic, this is a tool I found which works pretty well: https://github.com/ArchiveTeam/grab-site

Crawling is of course the other option. I've seen https://github.com/ArchiveTeam/grab-site in the wiki, but I'm unsure how to host the resulting .warc archives.

It's a completely different tool, but I like using grab-site https://github.com/archiveteam/grab-site . Try --wpull-args=--span-hosts='' or something to make it mirror all subdomains. It outputs in WARC format which can be read with a site like https://replayweb.page.

Thusly, as a bit of a statement, here's your "I will do it myself even if I have to bash my head against the wall" collection of the Developer Story on 10-20 top users. I know there are some blogs on old web design, perhaps it might be worth their while as a memento of an era bygone. And as for myself, I am looking into setting up a dedicated server for either grab-site or ArchiveBox. Possibly both!

I've had great success with using Joplin for this, with Syncthing as a sync backend. Works well across OSes; I use it on Linux, macOS, Windows and Android.

https://joplinapp.org/

The tools to manipulate SQL aren't that bad, no.

But rather than having a self explanatory markdown & flat file, now I have to start learning about the schema & making specific tools (in my preferred language) for manipulating Joplin's schema.

Suddenly I'm digging through 20 different technic specs to decode what data is where, how it works, and what I can do to it. Want to edit history? This is the best help you'll get, pray it's adequately technical to expedite you to your purpose: https://github.com/laurent22/joplin/blob/dev/readme/dev/spec...

As I began with, I struggle to imagine anything that generates anywhere near as much user agency as flat files and markdown. Having boring common data & systems lets me apply portable skills I already have, rather than having to skill up in some particular product's own ecosystem.

Joplin is a good open source option too, feels more like the original Evernote in terms of UI/UX https://github.com/laurent22/joplin/

Joplin, an open source, extendable, Markdown-based hierarchical note-taking app: https://joplinapp.org/

It lets you choose a synchronization backend, offers applications for every major desktop and mobile OS (also has a terminal version). You can create notebooks and subnotebooks to organize your notes. You can also add tags for better search experience. I created notebooks for specific domains (work-related, home improvement, etc.) and also keep a "temp" for quick notes and W.I.P. snippets.

Its only con that it uses Electron on desktop which causes relatively slow start of the application.

done