grub-2.0 VS spyglass

Not finished, but the Selenium based crawler works pretty well to combat most blocks: https://github.com/kordless/grub-2.0

For IP blocks, try this: https://github.com/kordless/mitta-screenshot

I started https://mitta.us as this, but am pivoting to prompt management for GPT-3. I've Open Sourced the code for the crawler here: https://github.com/kordless/grub-2.0. The entire system uses Google Vision for extracting text. I dislike fiddling with the DOM...

If you are interested in using Solr for this, I can provide instructions to you. I'm kordless at the gmails ... com.

Depending on the use case you might try imaging the page, then send the image to an ML model for full text before indexing. If you need links extracted, Selenium also supports parsing the assembled DOM: https://github.com/kordless/grub-2.0/tree/main/aperture

I’ve found imaging the page and doing OCR on the image is quite good for text extraction. Many pages on the Internet render with JavaScript, which means BS may not see the text in the DOM.

Here is the code to do some of that: https://github.com/kordless/grub-2.0

No direct experience but I saved some links about this:

https://wiby.me/about/guide.html

https://github.com/spyglass-search/spyglass

https://yacy.net/

Interesting thread(s) on HN about the last one:

https://news.ycombinator.com/item?id=32597309

I've been working on a self-hosted personal search app (https://github.com/spyglass-search/spyglass) and have recently been playing around with how to integrate it with local LLMs. I think this would be an awesome step into having your own personal assistant that can search through all your data and give you analysis / summaries.

We have an open-source desktop app (https://github.com/spyglass-search/spyglass) and are working on a web version that has more powerful features such as conversation search, check it out!

I'm one of the devs for Spyglass ([https://github.com/spyglass-search/spyglass](https://github.com/spyglass-search/spyglass)) an open-source personal search app. We're excited about all the advancements with language models recently and wanted to try merging the two ideas together to form something even better.

Another more recently available option is spyglass; it is more tenable than YaCy but dev is mostly on MacOS with focus on a desktop interface. I like the idea of web based interface.

I'm part of small team that's been building an open-source personal search engine (https://github.com/spyglass-search/spyglass). One of the major use cases is searching your local files & their contents which aligns perfectly with how Obsidian stores it notes.

I'm one of the developers of Spyglass (https://github.com/spyglass-search/spyglass), an open-source self-hosted personal search engine. We recently added the ability to search through your Reddit saved & upvoted posts!