linkwarden VS grab-site

Try Linkwarden - https://linkwarden.app

- Preserve bookmarks by capturing a screenshot of the saved page.

- Open-source and fully self-hostable.

- Support for collaborative bookmarking.

P.S. I’m the maintainer of the project.

This is one of the main reasons I created Linkwarden - an open-source collaborative bookmark manager to collect, organize and preserve webpages:

GitHub: https://github.com/linkwarden/linkwarden

Website: https://linkwarden.app

Someone posted this to HN a few days ago

https://linkwarden.app/

It looks very appealing, but I haven’t had a chance to try it myself just yet.

Great timing! Check this tool out: https://news.ycombinator.com/item?id=36942308

https://github.com/linkwarden/linkwarden

Linkwarden is a fully self-hostable, open-source collaborative bookmark manager to collect, organize and archive webpages.

Linkwarden was built using TypeScript and NextJS, backed by a PostgreSQL database for the lighter-weight data. The rest of the data can be chosen either to be stored on the filesystem, or stored on the cloud on Digital Ocean Space/AWS S3, the reason for the cloud storage solution was for the Cloud offering [1], we realized that the preserved webpages (archives) take up space pretty quickly and S3 was much more efficient for this task. On the front-end we used TailwindCSS for styling and Zustand for state management.

You could either use our Cloud offering (with 14-day free trial) to directly support this project and experience Linkwarden, or you could self-host it on your own machine and have maximum flexibility.

Also please make sure to visit/star our GitHub repo [2].

Feel free if you had any questions, we'll do our best to answer it.

[1]: https://cloud.linkwarden.app/register - Hosted in Digital Ocean's datacenter located here in Toronto, ON.

[2]: https://github.com/linkwarden/linkwarden

Some others I looked at: https://github.com/Kovah/LinkAce/ (PWA) https://github.com/sissbruecker/linkding https://github.com/ndom91/briefkasten (PWA) https://github.com/Daniel31x13/link-warden (PDF)

The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site (https://wiki.archiveteam.org/index.php/Who_We_Are) but I found that in less than 15 seconds of searching so do your own diligence.

You can use grab-site with --no-offsite-links and --igsets=mediawiki.

Does the forum provide real-time notification for new posts? Like maybe a RSS feed, or a 'New' section? If so, some scripting around grab-site or httrack could grab them quickly.

After a lot of searching for a similar topic, this is a tool I found which works pretty well: https://github.com/ArchiveTeam/grab-site

Crawling is of course the other option. I've seen https://github.com/ArchiveTeam/grab-site in the wiki, but I'm unsure how to host the resulting .warc archives.

It's a completely different tool, but I like using grab-site https://github.com/archiveteam/grab-site . Try --wpull-args=--span-hosts='' or something to make it mirror all subdomains. It outputs in WARC format which can be read with a site like https://replayweb.page.

Thusly, as a bit of a statement, here's your "I will do it myself even if I have to bash my head against the wall" collection of the Developer Story on 10-20 top users. I know there are some blogs on old web design, perhaps it might be worth their while as a memento of an era bygone. And as for myself, I am looking into setting up a dedicated server for either grab-site or ArchiveBox. Possibly both!