topzemen VS Internet-Places-Database

I created adult entertainment apps to organize and have fun with images and videos:

- RuGiVi: https://github.com/pronopython/rugivi - Browse your collection of images on an endless screen. Tested with more than 700.000 images at once!

- Fapel System: https://github.com/pronopython/fapel-system - Organize your adult images and videos by just using hardlinks and directories.

- TopZemen: https://github.com/pronopython/topzemen - Let the images float on your screen or rain down next to your browser window.

- Fplyr: https://github.com/pronopython/fplyr - An audio player to play moaning sounds in the background.

Everything for Ubuntu Linux and in parts also for Windows!

I agree with your points.

You might be interested in reddit webscraping thread https://www.reddit.com/r/webscraping/

My passion project is https://github.com/rumca-js/Django-link-archive

Currently I use only one thread for scraping, I do not require more. It gets the job done. Also I know too little to play more with python "celery" threads.

My project can be used for various things. Depends on needs. Recently I am playing with using it as a 'search engine'. I am scraping the Internet to find cool stuff. Results are in https://github.com/rumca-js/Internet-Places-Database. No all domains are interesting though.

https://github.com/rumca-js/Django-link-archive/blob/main/rs...

I know that there already are spiders, metadata processing packages for python, but I like having control over the process.

Old man yelling at the cloud. I hate also:

- blocking me with 403 because my user agent is not "mainstream". Why do I have to use chrome undetected to read some RSS feeds? Why can't I use third party clients? Contents can have adverts. I just want my own layout, buttons

- RSS feeds protected with cloudflare, so tools cannot read feeds easily

- not using, or outright blocking RSS functionality in wordpress. Some sites could be more open that way, but no. RSS feeds are closed/removed

- some sites have "/blog" location, but the main domain is empty, or nearly empty, or returns 404. Can I trust such location?

- when HTML meta data are not available. I like YouTube. It allows me to scrape metadata, but it protects video contents, and that is good

- weird redirects. Domain does not have any contents. Does not describe what it is. It just have javascript redirects. From main domain to some weird locations within the domain

- url shorteners, vanity links. You do not know where you will be transported. I understand they are counting sheep, but they sacrifice my security

- google returning links with syntax "https://www.google.com/url", not directly. Youtube does the same with syntax "https://www.youtube.com/redirect". For me again this is vulnerability

My ethic web scraper results are placed in: https://github.com/rumca-js/Internet-Places-Database.

I have been collecting personal sites for some time. Format JSON files. Links have tag 'personal'.

https://github.com/rumca-js/Internet-Places-Database

That is exactly why I scrape internet. I maintain all found domains in github:

https://github.com/rumca-js/Internet-Places-Database

I wanted to have all links about amiga, or commodore, chiptune.

It is not a search engine. For now, it is only data.

Maybe this will help somebody, or somebody will be able to use this data better.

I have a demo app running on rpi. It may be immediately broken if top many ppl accessed it.

https://renegat0x0.ddns.net/apps/places

Attention is limited. We cannot see everything on the Internet. We do not have enough time for that.

There is a lot of valuable and interesting data on the Internet, but it is not visible. Certainly high quality, low profile blog that ended its development in 2015 will not be ranked high in Google.

Media platform, search engines monetize content. YouTube channels need to churn new content every week or so to stay relevant and to stay watchable.

Our society produces content, not quality, not products.

SEO can be gamed, it is impossible to create objective index of valuable content. Bad actors will hack the game, spam results, destroy quality to gain profit.

Google search engine most often connects users with media sites, with news sites, with the middle men. The more often not connect users with product directly. Write "search engine" in search query, you may not only find search a "search engines" but articles about "Best search engines in 2024", or "best SEO tricks to boost your page".

Google does not have any incentive to fix this. Search engines are dead tech. It will be replaced by chatbots in a few years. People will not search for content, content will be generated at wish.

Some time ago I have created my own domain repository with domain names: https://github.com/rumca-js/Internet-Places-Database

I wanted to find "wargames" related pages. It is quite impossible to find anything interesting concerning warhammer on the normie internet (not Facebook).

The second thing is I cannot find anything "amiga" related.

This solved this my initial problem. I have also found out that many interesting pages are gone. I think that Google directing our attention toward "content" broke good quality pages.

Right now I am using less and less google, because I use more and more my bookmark manager.

https://github.com/rumca-js/Django-link-archive

My solutions may not be as complex as common crawl, but they are enough for me. For now. I am still working on my program. It has been fun and interesting experience for me, and I learned a lot. About open graph protocol, about schema, about web scraping, etc. etc. Maybe this will inspire people to be more self sufficient, and more self-hostable.

In times of walled gardens we need more standard, and more open data to keep what remains of the old wild west of the Internet.

Sorry for spamming, but I also create list.

In my repo maintain domains in JSON format. Tags "video game" and "open source" also provide list of open source games.

Other useful combination is tag "self-host", which provides self hostable programs, etc etc.

Link:

https://github.com/rumca-js/Internet-Places-Database

Some games are tagged "video game port" because they are reimplementation of existing games rather than providing something new.

I started gathering domains to see for myself the state of the Internet

https://github.com/rumca-js/Internet-Places-Database

I have many observations.

One is that I cannot see aby useful amiga links. I had to manually search them for some time. Some parts of the old internet exist, but are buried.

Second is that spam sites are everywhere. Not only AI generator.

Next is that personal sites exist, but they are often boring. Also 'CV sites' are a waste of time for me. I wonder how many of them are fake.

Many sites have poorly set up HTML meta fields, title, description. How anybody is supposed to find them?

I prefer going to passionate personal site about programming tips that reading content farms. It is difficult to find such sites.

You can find many RSS feeds, links in my repository

https://github.com/rumca-js/Internet-Places-Database/tree/ma...

It contains also domain lists, that include tag indicating, if it is personal, or not.

I am running my personal web crawler since September of 2022. I gather internet domains and assign them meta information. There are various sources of my data. I assign "personal" tag to any personal website. I assign "self-host" tag to any self-host program I find.

I have less than 30k of personal websites.

Data are in the repository.

https://github.com/rumca-js/Internet-Places-Database

I still rely on google for many things, or kagi. It is interesting to me, what my crawler finds next. It is always a surprise to see new blog, or forgotten forum of sorts.

This is how I discover real new content on the Internet. Certainly not by google which can find only BBC, or techcrunch.

Google will not be interested in fixing search. It also may not be possibile because of ai spam. They would like to invest in deep mind/bard/gemini than to fix technology that will be obsolete in a few years.

I have started scanning domains to see how many different places there are in the internet. Spoiler: Not many.

We could try to create curated open databases for links, forums, places, and links, but in ai era it will always be a niche.

Having said that I think that it is a good thing. If it is a niche it will not be spoiled by normal users expecting simple behavior, or corporations trying to control the output.

Start your blog

Start your curated lists of links.

Control your data. Share your data.

Link https://github.com/rumca-js/Internet-Places-Database