HN-Paywall-Archiver VS SingleFile

Source code: https://github.com/MostlyEmre/hn-anti-paywall

Now let me overexplain.

-Why?-

I never liked paywalled articles. I understand where they come from, but I don't like where we cross our paths.

This is why I don't use major news aggregators anymore. Instead, I spend my "catching-up-with-the-world-time" on Hacker News. However, Hacker News (HN) also has its fair-share of paywalled articles. (Around 11.6% according to my short-lived, half-assed attempt at measuring it. See my super old data https://hpa.emre.ca/ I tell the story below.)

-First try-

Around a year ago, when I ran the above experiment, my goal wasn't to run that experiment. It was during my self-teaching & career-changing process, I decided to build a React HN clone. To make it stand-out from the bunch, I added a paywall feature. It would detect paywalled articles and would add an archive URL into the metadata.

The issue with archiving is unless someone archived the link before on the {archiving-project} then the link is most likely not archived. So me sending people to those projects meant nothing. It kinda meant something for me from an ideological standpoint but I assume you are not me.

This rubbed me the wrong way. I decided to build a backend (See https://github.com/MostlyEmre/HN-Paywall-Archiver) that would scan the links and automatically to detect paywalls close to real-time and submit paywalled ones to archive.is for archival. I used Nodejs, Firebase, and React. I was -still am- really proud because I believed it was doing public good in terms of digital preservation. Only 1 person needed to run this script to benefit everyone. As an extra, I was curious on how many paywalled articles were being shared, by whom, at what time. So I also created some analytics functionality to gather the data. And later created a UI to present it.

HN-Paywall-Archiver was great but I stopped running the backend at some point. Because at that point couldn't find a way to continuously run my backend code on some platform for cheap or didn't try hard enough.

P.S. Recently I've been thinking of remaking this version with Cloudflare Workers.

-Hacker News Paywall Archiver Userscript-

After almost a year, I got into userscripts. Super great super awesome concept. People seem to hate javascript unless it is presented as a userscript. So I decided to get my hands dirty to create a simple solution that solves the paywall issue on HN without breaking any hearts.

My solution is not perfect as it had to be simple. But here's the rundown.

Pros:

- Does not beg for attention.

That's interesting. I have been saving articles as PDF files, which is browser-independent, but useful just for search and reference, a nuisance to quote/copy-and-paste.

If I search only the computer, I don't get results from EBay and Amazon at the top. The idea of keeping the knowledge base separate from the primary notes is a good idea. In my case, that knowledge base is the file system, and the primary notes are whatever I choose.

When I was using Evernote, the inbox was the knowledge base and notebooks were the focus. I just had too many different potential projects going on to manage this well.

Looking to focus.

I'll revisit Firefox and SingleFile.

Explanation of the zip file inside.

https://github.com/gildas-lormeau/SingleFile/blob/master/faq...

[2] https://github.com/gildas-lormeau/SingleFile/blob/master/faq...

I agree it would be "great" a complete website in the ZIP. I think this is technically possible, someone just have to code it.

[1] https://github.com/gildas-lormeau/SingleFile#singlefile

Singlefile [1] works pretty well for me for that use case.

It has the added advantage that the file format is just plain HTML, and together with “reader mode” in most browsers, it’s a great way to save long-form text or other mostly static pages for later reference.

It obviously doesn’t work for very dynamic pages, let alone web apps.

[1] https://github.com/gildas-lormeau/SingleFile

I’ve tried all the third party services for archiving interesting things over the years but nothing beats saving everything to your local filesystem using [SingleFile](https://github.com/gildas-lormeau/SingleFile) and using a full-text search front over the directory (something like Houdahspot, for example).

If you want to save them as a file locally you could use something like SingleFile. You could also put the URL for each wiki into archive.org's Save Page Now so that anyone can access it. Either way, without scripting, you'll have to do some manual labor to get the URL for each wiki.

For small numbers of pages, the SingleFile[0] extension for Firefox (WebExtension) is pretty handy. It's not "archival quality", though, if that's the kind of "archiving" you're doing.

[0] https://github.com/gildas-lormeau/SingleFile