cockroach VS Trino

Employ distributed databases like CockroachDB, Cassandra, or managed services like Amazon Aurora Global Databases.

In fact, even in the absence of these commercial databases, users can effortlessly install PostgreSQL and leverage its built-in pgvector functionality for vector search. PostgreSQL stands as the benchmark in the realm of open-source databases, offering comprehensive support across various domains of database management. It excels in transaction processing (e.g., CockroachDB), online analytics (e.g., DuckDB), stream processing (e.g., RisingWave), time series analysis (e.g., Timescale), spatial analysis (e.g., PostGIS), and more. For non-professional users seeking to explore vector databases, they can readily download the open-source PostgreSQL or utilize managed services like Supabase and Neon to establish their own basic AI applications. Other than PostgreSQL, several open-source databases, including OpenSearch, ClickHouse, and Cassandra, have implemented their own vector search functionality. You do not need to adopt a new vector database if you have already used these systems.

Postgres-compatible DSQL solutions: Distributed SQL databases that are either Postgres or Postgres-compatible like Citus, Aurora DSQL, and CockroachDB might be the right approach for handling your scale without forcing you to change much of your code. They each have their strengths and weaknesses and the right one for you depends on your use cases and needs.

Still very much "open-source": https://github.com/cockroachdb/cockroach

But relicensed to the "CockroachDB Software License" as a form of BSL to prevent reselling.

code in a given file is licensed under the BSL and ..." That is sucky.

[0] https://github.com/cockroachdb/cockroach?tab=License-1-ov-fi...

CockroachDB — Distributed SQL database built for cloud applications.

CockroachDB is a distributed SQL database designed for cloud applications. It provides strong consistency, horizontal scalability, and high availability.

Traditional databases — PostgreSQL, MySQL, etc. — store their data in proprietary formats. That format is optimized for that engine and can’t be directly accessed by anything else. Even if something like Trino can connect to Postgres, it’s still running queries through Postgres itself, not reading its storage directly. You’re just a client.

Trino: Trino (formerly known as PrestoSQL) is a high-performance distributed SQL query engine designed for data analysis. It offers efficient querying capabilities across multiple data sources, including various file formats, databases, and data lakes. These are some interesting background story between Trino and Presto: Presto was the original name of the project, and it was developed by Facebook. In December 2020, a significant portion of the Presto community decided to fork the project and renamed it Trino. Read more here: Trino Blog.

However, Iceberg defines the storage format, leaving the complexities of data ingestion and processing, especially for real-time streams, to separate systems. While query engines like Trino or Athena excel with static datasets, they aren't designed for continuous, low-latency ingestion and transformation of streaming data into Iceberg. This often forces engineers to integrate multiple complex tools, increasing operational overhead and fragility.

By the way, I wanted to continue to use the previous experiment with Flink SQL and Iceberg, but I found out Trino doesn't support Iceberg's DynamoDB catalog. Therefore, I had to create a new one.

We decided not to waste time reinventing the wheel and simply installed Trino on our servers. It’s a full featured SQL query engine that works on your data. Now our analysts can use it to work with data from AppMetr and execute queries at different levels of complexity.

DevRel for StarRocks. Trino doesn't have a great caching layer (https://github.com/trinodb/trino/pull/16375) and performance (https://github.com/trinodb/trino/issues/14237) and https://github.com/oap-project/Gluten-Trino. In benchmarks and community user testing, StarRocks has outperformed.

What if my SQL engine is Presto, Trino [1], or a similar query engine? If it's federating multiple source databases we peel the SQL back and get... SQL? Or you peel the SQL back and get... S3 + Mongo + Hadoop? Junior analysts would work at 1/10th the speed if they had to use those raw.

[1] https://trino.io/