grab-site VS docker-swag

The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site (https://wiki.archiveteam.org/index.php/Who_We_Are) but I found that in less than 15 seconds of searching so do your own diligence.

You can use grab-site with --no-offsite-links and --igsets=mediawiki.

Does the forum provide real-time notification for new posts? Like maybe a RSS feed, or a 'New' section? If so, some scripting around grab-site or httrack could grab them quickly.

After a lot of searching for a similar topic, this is a tool I found which works pretty well: https://github.com/ArchiveTeam/grab-site

Crawling is of course the other option. I've seen https://github.com/ArchiveTeam/grab-site in the wiki, but I'm unsure how to host the resulting .warc archives.

It's a completely different tool, but I like using grab-site https://github.com/archiveteam/grab-site . Try --wpull-args=--span-hosts='' or something to make it mirror all subdomains. It outputs in WARC format which can be read with a site like https://replayweb.page.

Thusly, as a bit of a statement, here's your "I will do it myself even if I have to bash my head against the wall" collection of the Developer Story on 10-20 top users. I know there are some blogs on old web design, perhaps it might be worth their while as a memento of an era bygone. And as for myself, I am looking into setting up a dedicated server for either grab-site or ArchiveBox. Possibly both!

I have a NAS on .0.181 and a swag container (on a different port than nginx) on .0.180 that points to my public facing services. For obvious reasons, I don't want my public domain to point to any other ports/addresses on my home network. Additionally, as elegant as swag is, it requires authentication and so won't work for simple local DNS. I now have one local domain for each server and an nginx instance on each that resolves to my different services on each.

OP is even linking the Github... https://github.com/linuxserver/docker-swag

I have an UnRAID server with a few services (Jellyfin, Nextcloud, etc.) running on it behind Linux Servers' SWAG reverse proxy container, which is built on Nginx and Let's Encrypt. This is pointed to a DuckDNS link, which is then pointed at my domain with a CNAME. So I can access Jellyfin, for example, at jellyfin.mydomain.com. A few weeks ago, due to seemingly unrelated issues, I got a new modem/router, an Arris SURFboard G34. For the first few weeks, everything was working as before. But now, when on my LAN, I can't get to my services at the proxied domain. It times out every time. There are no errors in SWAG's logs, nothing seems amiss in the router's web interface, and the services are available both at their IP:port address and, when not on my LAN, I can access them at the domain no problem.

Reverse proxy, probably? I use Docker SWAG, setup here, with DuckDNS and it works really well for me. There are of course many ways to reverse proxy, as I linked to earlier.