grub-2.0 VS sonic

Not finished, but the Selenium based crawler works pretty well to combat most blocks: https://github.com/kordless/grub-2.0

For IP blocks, try this: https://github.com/kordless/mitta-screenshot

I started https://mitta.us as this, but am pivoting to prompt management for GPT-3. I've Open Sourced the code for the crawler here: https://github.com/kordless/grub-2.0. The entire system uses Google Vision for extracting text. I dislike fiddling with the DOM...

If you are interested in using Solr for this, I can provide instructions to you. I'm kordless at the gmails ... com.

Depending on the use case you might try imaging the page, then send the image to an ML model for full text before indexing. If you need links extracted, Selenium also supports parsing the assembled DOM: https://github.com/kordless/grub-2.0/tree/main/aperture

I’ve found imaging the page and doing OCR on the image is quite good for text extraction. Many pages on the Internet render with JavaScript, which means BS may not see the text in the DOM.

Here is the code to do some of that: https://github.com/kordless/grub-2.0

Sonic - a project written in Rust, uses custom network communication protocol for fast communication between the client and the server.

This is uncanny, I just discovered ArchiveBox earlier today and set up a self-hosted instance on some home hardware for a collection of bookmarks of useful guides, tutorials, and references I've collected over the years.

Setting it up on K8s with sonic [1] as the search backend and importing a few hundred URLs only took ~an hour or so, and the cached pages look great for the most part.

[1] https://github.com/valeriansaliou/sonic

If you don't need advanced search features, you can use Sonic (https://github.com/valeriansaliou/sonic). It's blazing fast and you can save lot of money on servers.

What about Sonic? Maybe it requires a bit of integration, but it's simple and blazing fast.

git clone https://github.com/valeriansaliou/sonic.git