orjson VS datasette

The library already has support for an HTTP client that allows bypassing Cloudflare - CurlImpersonateHttpClient. Since we have to work with JSON responses we could use parsel_crawler added in version 0.3.0, but I think this is excessive for such tasks, besides I like the high speed of orjson.. Therefore, we'll need to implement our crawler rather than using one of the ready-made ones.

(https://github.com/ijl/orjson). In this implementation, the same JSON

orjson – Fast, correct Python JSON library

orjson is the fastest JSON library available for python. It natively manages dataclass objects, datetime, numpy and UUID objects.

As we began venturing down that road, a few things emerged that would constitute significant changes to some of the core parts of Litestar, but there were two things in particular that started a chain reaction of changes by opening up further possibilities: The new DTOs and our switch from orjson to msgspec.

A couple of reasons which pop to mind:

- Searching a plain text data file is O(n). Searching a SQLite database that has been properly indexed, which is very easy to do nowadays with FTS5, is O(log n) worst case scenario and O(1) in the best case. This doesn't explain why SQLite over a dataframe or anything, but it definitely justifies it over plain text for large email collections.

- SQLite is really easy to write custom views and programs around. Virtually every major programming language can work with it without issue. See also: simonw's wonderful https://datasette.io/ .

- SQLite is an accepted archival format by the Library of Congress, if you ever want to go down the rabbit hole of digital preservation.

It's https://datasette.io - I'm still having so much fun with it, especially since any idea I want to experiment with can be justified as a Datasette plugin!

SQLite is used because it's lightweight, requires no server setup, and provides a self-contained database solution ideal for this type of data collection. Additionally, Datasette can be used to easily query, visualize, and publish the data for later analysis.

It does seem like a job well suited to simonw's https://github.com/simonw/datasette

In theory, you could make it happen with:

- a db

- a db browser

- a {bookmarks,browser,etc}-to-db tool

For instance,

- sqlite

- datasette https://datasette.io/

- pocket-to-sqlite https://datasette.io/tools/pocket-to-sqlite

Furthermore, you could integrate e.g. Claude with this, by creating a lightweight model-context-protocol server that lets Claude browse through datasette. You would probably want to throw in a vector db in-between somewhere, so you could get semantic search over bookmarks to work nicely.

Seems like a perfect job for Datasette: SQLite plus web api and UI

“Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.”

https://datasette.io/

You might like this: https://datasette.io/ and it might even fill your blogging needs. Fully opensource (and there are extension to use llm's with it).

I wrote an async wrapper around SQLite in Python - I'm using a thread pool: https://github.com/simonw/datasette/blob/main/datasette/data...

I have multiple threads for reads and a single dedicated thread for writes, which I send operations to via a queue. That way I avoid ever having two writes against the same connection at the same time.