what-hn-says-webext VS hackernews-button

Some extensions like this [0] don't work at all outside exact URL matches. Are there extensions or website to find similar webpages by relative text analysis or NLP?

[0] https://github.com/pinoceniccola/what-hn-says-webext

Similar extension for Hacker News discussions - https://github.com/pinoceniccola/what-hn-says-webext

It's worth noting that my extension is far from perfect – it turns out that determining whether a specific page has been submitted to Hacker News is far from a trivial problem to solve. In general, this is because multiple URLs can map to the same page.

Direct string comparison of the current URL to previously submitted ones doesn't work because there are many ways for two identical web pages to have different URLs. For example, the URL fragments can differ (the part after the "#" that may or may not be present). Also there can be tracking parameters (often—but not necessarily—prefixed with "utm_"), which don't change anything about the page. But the URL parameters can't be entirely disregarded because sometimes sites, forums in particular, rely on them – consider pages that use an "?id=..." parameter for different pages. Thus some parameters should be removed, but some shouldn't. The same website having different domains (or domains that change over time) further complicates the situation.

My solution was to "canonicalize" URLs by transforming them into a simplified form using some pretty rough heuristics for common sources of noise. The Python code to do that is here: https://github.com/jstrieb/hackernews-button/blob/master/can...

All of this to say that even though I've used my extension for months and have been quite happy, there will inevitably be false negatives.

Thanks for clearing that up, yes I'm not that familiar with Bloom filters, seems like an interesting and useful concept. It could probably (pun intended) be applied to many applications to increase privacy.

I like the [1] Workflow file you've made, the comments really help with reading shell code. I'm also amazed you can query 4M entries everyday with BigQuery, I thought that might be fairly expensive to do right? Or is this below a free tier?

[1] https://github.com/jstrieb/hackernews-button/actions/runs/61...