Conkey VS Internet-Places-Database

Most of my programs were written for my own use, including:

• A keyboard layout to type numerous non-English letters, punctuation marks and mathematical symbols, originally for Windows but subsequently ported to Linux and Mac [https://github.com/bradrn/Conkey]

• A ‘sound change applier’ for my hobby of language construction, to simulate the process of historical sound change [https://bradrn.com/brassica/]

• A small browser extension to save the full text of all webpages I visit, and a local client to search the database [not open-sourced, apologies!]

The first two have gained a few other users since being released, but I’m pretty sure I’m still the one who uses them the most!

I made my own crossplatform multilingual layout [0]. Although it’s based on QWERTY, it shouldn’t be hard to remap the Linux and Mac versions to any other base layout, since they’re autogenerated from the Windows version.

[0] https://github.com/bradrn/Conkey

The biggest one for me is undoubtedly my custom keyboard layout Conkey [0], which I use constantly (including for typing this very comment). I hate the way the base US layout tends to get distorted in other keyboard layouts with good support for non-ASCII characters, so Conkey had the explicit goal of retaining that basic unshifted layout. I’ve also ended up porting Conkey to Mac and Linux — and given that I’m slowly switching from Windows to Linux, at least the Linux ports have ‘scratched my own itch’ too, which is nice.

Also, I made a utility to archive the full text of every website I view and store it in a SQLite database for searching. It’s proven pretty useful when I want to find something I saw a while ago and then forgot. (I haven’t attempted to open-source it, though — it consists of three entirely separate components, two of which were a pain to set up. I must try to get it into a more usable state one of these days.)

What else… my sound change applier [1], perhaps? Not that I use it very much, because I only need it on those occasions when I want to do some conlanging, which I haven’t had much time for recently. Actually, sound change appliers strike me as being very much a ‘scratch own itch’ type of project in general… sometimes it feels like every conlanger has written their own, and no two can agree on a nice design. Everyone just has their own unique preferred way of doing things.

[0] https://github.com/bradrn/Conkey

[1] https://github.com/bradrn/brassica

For a ~50-year-old program, Emacs’s support for multilingual input — and really, it’s all-round flexibility — continually amazes me! For myself I prefer my own custom keyboard layout [0], because it works outside Emacs too, but I’d happily use Emacs’s own input methods if that would be sufficient.

(In fairness, I have found one weak spot, namely font support… I’ve used ‘unicode-fonts’ [1] with some success, but reportedly it doesn’t work with the latest Emacs. Ah well, it’s at least fairly rare that this becomes a problem in practice.)

[0] https://github.com/bradrn/Conkey

[1] https://github.com/rolandwalker/unicode-fonts

To port my keyboard layout [0] to OSX, I used ‘osxkb’ [1], which outputs an OSX keyboard layout bundle given a simple textual specification file. It was originally created specifically to port Conkey to OSX, but should be entirely usable for other purposes as well.

[0] https://github.com/bradrn/Conkey

> But instead, we're dealing with Latex - a language that overcomplicates the most basic features such as fonts, tables and special characters.

I can’t really argue with the rest of your post, but in my experience this is incorrect. Fonts and special characters are both trivial if you use XeTeX, and tables, though slightly clumsy, are still pretty easy. As an example, see the documentation I wrote for https://github.com/bradrn/Conkey, which makes extremely heavy use of all three features. (As documentation for a keyboard layout, it uses characters from pretty much every corner of Unicode, and accompanying tables of many shapes and sizes to show how to type these characters; I needed to use Gentium in order to render all these characters, with Times New Roman as a fallback. I found that LaTeX could ably handle all of these complecations.)

I started gathering domains to see for myself the state of the Internet

https://github.com/rumca-js/Internet-Places-Database

I have many observations.

One is that I cannot see aby useful amiga links. I had to manually search them for some time. Some parts of the old internet exist, but are buried.

Second is that spam sites are everywhere. Not only AI generator.

Next is that personal sites exist, but they are often boring. Also 'CV sites' are a waste of time for me. I wonder how many of them are fake.

Many sites have poorly set up HTML meta fields, title, description. How anybody is supposed to find them?

I prefer going to passionate personal site about programming tips that reading content farms. It is difficult to find such sites.

You can find many RSS feeds, links in my repository

https://github.com/rumca-js/Internet-Places-Database/tree/ma...

It contains also domain lists, that include tag indicating, if it is personal, or not.

I am running my personal web crawler since September of 2022. I gather internet domains and assign them meta information. There are various sources of my data. I assign "personal" tag to any personal website. I assign "self-host" tag to any self-host program I find.

I have less than 30k of personal websites.

Data are in the repository.

https://github.com/rumca-js/Internet-Places-Database

I still rely on google for many things, or kagi. It is interesting to me, what my crawler finds next. It is always a surprise to see new blog, or forgotten forum of sorts.

This is how I discover real new content on the Internet. Certainly not by google which can find only BBC, or techcrunch.

Google will not be interested in fixing search. It also may not be possibile because of ai spam. They would like to invest in deep mind/bard/gemini than to fix technology that will be obsolete in a few years.

I have started scanning domains to see how many different places there are in the internet. Spoiler: Not many.

We could try to create curated open databases for links, forums, places, and links, but in ai era it will always be a niche.

Having said that I think that it is a good thing. If it is a niche it will not be spoiled by normal users expecting simple behavior, or corporations trying to control the output.

Start your blog

Start your curated lists of links.

Control your data. Share your data.

Link https://github.com/rumca-js/Internet-Places-Database

There are already many project about search:

- https://www.marginalia.nu/

- https://searchmysite.net/

- https://lucene.apache.org/

- elastic search

- https://presearch.com/

- https://stract.com/

- https://wiby.me/

I think that all project are fun. I would like to see one succeeding at reaching mainstream level of attention.

I have also been gathering links meta data for some time. Maybe I will use them to feed any eventual self hosted search engine, or language model, if I decide to experiment with that.

- domains for seed https://github.com/rumca-js/Internet-Places-Database

- bookmarks seed https://github.com/rumca-js/RSS-Link-Database

- links for year https://github.com/rumca-js/RSS-Link-Database-2024

I have myself dabbled a little bit in that subject. Some of my notes:

- some RSS feeds are protected by cloudflare. It is true however that it is not necessary for majority of blogs. If you would like to do more then selenium would be a way to solve "cloudflare" protected links

- sometimes even selenium headless is not enough and full blown browser in selenium is necessary to fool it's protection

- sometimes even that is not enough

- then I started to wonder, why some RSS feeds are so well protected by cloudflare, but who am I to judge?

- sometimes it is beneficial to cover user agent. I feel bad for setting my user agent to chrome, but again, why RSS feeds are so well protected?

- you cannot parse, read entire Internet, therefore you always need to think about compromises. For example I have narrowed area of my searches in one of my projects to domains only. Now I can find most of the common domains, and I sort them by their "importance"

- RSS links do change. There need to be automated means to disable some feeds automatically to prevent checking inactive domains

- I do not see any configurable timeout for reading a page, but I am not familiar with aiohttp. Some pages might waste your time

- I hate that some RSS feeds are not configured properly. Some sites do not provide a valid meta "link" with "application/rss+xml". Some RSS feeds have naive titles like "Home", or no title at all. Such a waste of opportunity

My RSS feed parser, link archiver, web crawler: https://github.com/rumca-js/Django-link-archive. Especially interesting could be file rsshistory/webtools.py. It is not advanced programming craft, but it got the job done.

Additionally, in other project I have collected around 2378 of personal sites. I collect domains in https://github.com/rumca-js/Internet-Places-Database/tree/ma... . These files are JSONs. All personal sites have tag "personal".

Most of the things are collected from:

https://nownownow.com/

https://searchmysite.net/

I wanted also to process domains from https://downloads.marginalia.nu/, but haven't got time to read structure of the files

On the other hand it is not 1995. Time has moved on. I wrote a Simple RSS feed, that also serves as search engine for bookmarks.

I am able to run it in attick on raspberry pi. We do not have to rely so heavily on google.

https://github.com/rumca-js/Django-link-archive

It is true that it does not serve me as google, or kagi replacement. It is a very nice addition though.

With a little bit off determination I do not have to be so dependent on google.

Here is also a dump of known domains. Some are personal.

https://github.com/rumca-js/Internet-Places-Database

...and my bookmarks

https://github.com/rumca-js/RSS-Link-Database

Some more years, and google can go to hell.

[4] https://github.com/rumca-js/Django-link-archive

These are exported then to github repositories:

[5] https://github.com/rumca-js/RSS-Link-Database - bookmarks

[6] https://github.com/rumca-js/RSS-Link-Database-2023 - 2023 year news headlines

[7] https://github.com/rumca-js/Internet-Places-Database - all known to me domains, and RSS feeds

My own repositories:

- bookmarked entries https://github.com/rumca-js/RSS-Link-Database

- mostly domains https://github.com/rumca-js/Internet-Places-Database

- all 'news' from 2023 https://github.com/rumca-js/RSS-Link-Database-2023

I am using my own Django program to capture and manage links https://github.com/rumca-js/Django-link-archive.