MongoDB VS LiteDB

For larger datasets or ongoing scraping, cloud-based solutions like MongoDB, Amazon S3 or Apify Storage become necessary. They’re designed to handle large volumes of data and offer quick querying capabilities.

MongoDB: Set up a MongoDB database (you can use MongoDB Atlas for a cloud-based option).

If you don’t have an account, you can sign up for a free account at the MongoDB site. Create an organization and customize the name for your purposes, but we will be calling it My-Project for this example.

MongoDB: A NoSQL database for storing original URLs.

MongoDB: A distributed file storage database.

Frequently, I consider MongoDB, especially the serverless version. It’s a robust database that benefits from the relational model and is a powerful NoSQL database with many features that most storages don’t provide.

(At first we're using a PostgreSQL image from the official DockerHub repository, but different databases such as MySQL/MariaDB or MongoDB will follow a similar pattern)

My first contact with Testcontainers was in a project in Golang and I liked so much the facility of provisioning MongoDB container to do the repository tests and after that, I decided to do same thing in my personal project in PHP using Symfony Framework.

3. MongoDB

Personally, I used to use LiteDB which is a NoSQL version which in v4 paired very nicely with F# thanks to Zaid's lovely LiteDB.FSharp library. Sadly, when v5 showed up, a lot of the F# niceties were lost given how the API was changed and v5 was not very F# friendly. You can still use it of course but you fall back to more unsafe F# code which is not ideal.

Maybe you want LiteDB? I don't think you need to write SQL, you can interact with the API.

Are you aware of LiteDB? It seems like your project is going for a very similar niche. Most people looking for this type of thing will probably go for the more mature and feature-rich solution (LiteDB). So if your project has some unique value proposition to distinguish it from LiteDB, you should elaborate on that.

Please consider using the primary key to generate an unique value on the db side.

LiteDB

Before checking this out, people might want to take a look through the issues and pull requests of which there are 500+ and 50+ respectively [1]. I was really optimistic about this project and it was headed in a great direction, but it's not in a production ready state, and it seems that the main guy behind it has decided to move onto other things. It's been about a year since there was any significant activity.

I just mention this because a lot of these little issues might only become more apparent after integrating the db into your project and so it can be a bit annoying. I ended up swapping to Linq2DB [1]. It's something, more or less, similar offering an ORM/LINQ type system as well as the ability to also use direct SQL if desired. But the neat thing is that it also uses a standardized API for the LINQ query language, so you can do things like swap from SQLite to PostgreSQL in one* line of code, so long as you're not using any provider specific extensions.

[1] - https://github.com/mbdavid/LiteDB

[2] - https://github.com/linq2db/linq2db

A mature NoSQL embedded/flatfile database like LiteDB would be nice. There are some similar Rust libraries but they aren't very close to production ready and the API tends to not be too user friendly. I had trouble finding one of these for a small app I had to write recently.