webring VS Django-link-archive

I like this author's idea of curating bookmark lists, but I think they are most effective when two criteria are followed: (1) keep the list small, (2) write small notes about each entry.

For example, the bookmarks list the author links to (https://www.marginalia.nu/links/bookmarks/) has 48 URLs annotated only by category. That's too many for my tiny brain to handle and I move on.

A webring like Hundred Rabbits' (https://webring.xxiivv.com/) has 203 entries. For me, this is in the same category as 48. (It also reminds me of those "Awesome X" lists on GitHub that end up flooded with hundreds of links.)

To attempt an example of what I mean, here's the bookmark list I publish on my website:

- Bret Victor (http://worrydream.com/) • interaction and abstraction

We're out here, just not in your workplace. Poke around in more decentralized tech spaces like federated social media and self-hosting groups. Talk to free software people, find local users' groups. As far as non-profits go, you may be interested in the EFF or the Internet Archive. Phrack Magazine (http://phrack.org/) is a very-long-running webzine for hackers, although it's been updating less and less frequently, and there are lots and lots and lots of webrings of little self-hosted sites which lean leftist. (see: https://webring.xxiivv.com/) There are also some in-person tech collectives like Cyberia: https://cyberia.club/ As far as individuals go, I don't believe Phineas Fisher has been caught yet, and of course there's Maia Arson Crimew (https://maia.crimew.gay/) who leaked the no-fly list.

XXIIVV Webring - https://github.com/XXIIVV/webring

I agree with your points.

You might be interested in reddit webscraping thread https://www.reddit.com/r/webscraping/

My passion project is https://github.com/rumca-js/Django-link-archive

Currently I use only one thread for scraping, I do not require more. It gets the job done. Also I know too little to play more with python "celery" threads.

My project can be used for various things. Depends on needs. Recently I am playing with using it as a 'search engine'. I am scraping the Internet to find cool stuff. Results are in https://github.com/rumca-js/Internet-Places-Database. No all domains are interesting though.

https://github.com/rumca-js/Django-link-archive/blob/main/rs...

Some things more project specific are in

Attention is limited. We cannot see everything on the Internet. We do not have enough time for that.

There is a lot of valuable and interesting data on the Internet, but it is not visible. Certainly high quality, low profile blog that ended its development in 2015 will not be ranked high in Google.

Media platform, search engines monetize content. YouTube channels need to churn new content every week or so to stay relevant and to stay watchable.

Our society produces content, not quality, not products.

SEO can be gamed, it is impossible to create objective index of valuable content. Bad actors will hack the game, spam results, destroy quality to gain profit.

Google search engine most often connects users with media sites, with news sites, with the middle men. The more often not connect users with product directly. Write "search engine" in search query, you may not only find search a "search engines" but articles about "Best search engines in 2024", or "best SEO tricks to boost your page".

Google does not have any incentive to fix this. Search engines are dead tech. It will be replaced by chatbots in a few years. People will not search for content, content will be generated at wish.

Some time ago I have created my own domain repository with domain names: https://github.com/rumca-js/Internet-Places-Database

I wanted to find "wargames" related pages. It is quite impossible to find anything interesting concerning warhammer on the normie internet (not Facebook).

The second thing is I cannot find anything "amiga" related.

This solved this my initial problem. I have also found out that many interesting pages are gone. I think that Google directing our attention toward "content" broke good quality pages.

Right now I am using less and less google, because I use more and more my bookmark manager.

https://github.com/rumca-js/Django-link-archive

My solutions may not be as complex as common crawl, but they are enough for me. For now. I am still working on my program. It has been fun and interesting experience for me, and I learned a lot. About open graph protocol, about schema, about web scraping, etc. etc. Maybe this will inspire people to be more self sufficient, and more self-hostable.

In times of walled gardens we need more standard, and more open data to keep what remains of the old wild west of the Internet.

I am running my of YouTube front-end.

Link: https://github.com/rumca-js/Django-link-archive

Demo: https://renegat0x0.ddns.net/apps/catalog

Allows me to add channels, download individual videos, bookmark videos, etc. Uses iframe to display the video.

I have no problems with viewing videos:

- I use ff with adblock

- videos are embedded using no-referrer-when-downgrade policy

I have literally no ads.

There are some drawbacks:

- YouTube may change their policies, change embedding strategies

- To have control over videos, you still have to manually download them

I wrote bookmark manager in Django.

https://github.com/rumca-js/Django-link-archive

You can self host it.

You can add RSS sources and auto import new links regularly.

It may not be stare of the art, but gets the job done.

Demo below, but may not be working when you look at it. It runs on raspberry pi.

https://renegat0x0.ddns.net/apps/catalog/entry/11503/

I have myself dabbled a little bit in that subject. Some of my notes:

- some RSS feeds are protected by cloudflare. It is true however that it is not necessary for majority of blogs. If you would like to do more then selenium would be a way to solve "cloudflare" protected links

- sometimes even selenium headless is not enough and full blown browser in selenium is necessary to fool it's protection

- sometimes even that is not enough

- then I started to wonder, why some RSS feeds are so well protected by cloudflare, but who am I to judge?

- sometimes it is beneficial to cover user agent. I feel bad for setting my user agent to chrome, but again, why RSS feeds are so well protected?

- you cannot parse, read entire Internet, therefore you always need to think about compromises. For example I have narrowed area of my searches in one of my projects to domains only. Now I can find most of the common domains, and I sort them by their "importance"

- RSS links do change. There need to be automated means to disable some feeds automatically to prevent checking inactive domains

- I do not see any configurable timeout for reading a page, but I am not familiar with aiohttp. Some pages might waste your time

- I hate that some RSS feeds are not configured properly. Some sites do not provide a valid meta "link" with "application/rss+xml". Some RSS feeds have naive titles like "Home", or no title at all. Such a waste of opportunity

My RSS feed parser, link archiver, web crawler: https://github.com/rumca-js/Django-link-archive. Especially interesting could be file rsshistory/webtools.py. It is not advanced programming craft, but it got the job done.

Additionally, in other project I have collected around 2378 of personal sites. I collect domains in https://github.com/rumca-js/Internet-Places-Database/tree/ma... . These files are JSONs. All personal sites have tag "personal".

Most of the things are collected from:

https://nownownow.com/

https://searchmysite.net/

I wanted also to process domains from https://downloads.marginalia.nu/, but haven't got time to read structure of the files

"""

Disclaimer: I have a automated RSS reader enabled in my network: https://github.com/rumca-js/Django-link-archive

On the other hand it is not 1995. Time has moved on. I wrote a Simple RSS feed, that also serves as search engine for bookmarks.

I am able to run it in attick on raspberry pi. We do not have to rely so heavily on google.

https://github.com/rumca-js/Django-link-archive

It is true that it does not serve me as google, or kagi replacement. It is a very nice addition though.

With a little bit off determination I do not have to be so dependent on google.

Here is also a dump of known domains. Some are personal.

https://github.com/rumca-js/Internet-Places-Database

...and my bookmarks

https://github.com/rumca-js/RSS-Link-Database

Some more years, and google can go to hell.

[4] https://github.com/rumca-js/Django-link-archive

These are exported then to github repositories:

[5] https://github.com/rumca-js/RSS-Link-Database - bookmarks

[6] https://github.com/rumca-js/RSS-Link-Database-2023 - 2023 year news headlines

[7] https://github.com/rumca-js/Internet-Places-Database - all known to me domains, and RSS feeds