python-fake-data-producer-for-apache-kafka VS ClickHouse

Aiven offers a range of open-source managed data infrastructures, including PostgreSQL, Apache Kafka, Elasticsearch, Grafana, InfluxDB, MySQL, Redis and more.

For the database I used Valkey on Aiven for free.

For this demo I'm using Aiven, they offer a 1GB free account, which is more than enough for a POC or prototype.

Unlike Neon, Aiven provides many open-source managed data infrastructures, such as PostgreSQL Apache Cassandra, Apache Kafka, Apache Kafka Connect, Apache Kafka MirrorMaker 2, Elasticsearch, Grafana, InfluxDB, M3, M3 Aggregator, MySQL, and Redis. It’s a jack of all trades regarding its support for open-source technology.

A PostgreSQL database (created using Aiven and connected using DBeaver).

Aiven

Aiven provides managed cloud service for PostgreSQL, making sure databases run smoothly, safely, and can grow easily on different cloud platforms.

An Aiven account to provision the Redis database.

I have the luxury of working for Aiven which provides professionals an integrated platform for all their data needs. In the last three years I witnessed the growth of the platform and its evolution with the clear objective to make it better usable at scale. Tooling like integrations, Terraform providers and the Console facilitate the work that platform administrators have to perform on daily basis.

I had good experience with Aiven in the past, we needed something located in the EU: https://aiven.io/

At Agnostic, we build open-source infrastructure for collaborative blockchain data platforms. One of our flagship tools is clickhouse-evm, a suite of high-performance User Defined Functions (UDFs) that brings native Ethereum decoding and querying capabilities directly into ClickHouse.

Over the past few weeks, I’ve been diving into ClickHouse — and it’s been full of surprises.

I attended the AWS Summit 2025 in Singapore. I enjoyed the event. There were booths from various companies which I found interesting, such as GitLab and ClickHouse. More importantly, I got to meet very interesting people.

https://github.com/ClickHouse/clickhouse

Disclosure: I started at Citus & ended up at ClickHouse

ClickHouse: A fast columnar database. It will be our final destination for clean data. And, for simplicity in this tutorial, we'll cleverly use it as our "memory" or state store to remember which events we've already seen recently.

In fact, even in the absence of these commercial databases, users can effortlessly install PostgreSQL and leverage its built-in pgvector functionality for vector search. PostgreSQL stands as the benchmark in the realm of open-source databases, offering comprehensive support across various domains of database management. It excels in transaction processing (e.g., CockroachDB), online analytics (e.g., DuckDB), stream processing (e.g., RisingWave), time series analysis (e.g., Timescale), spatial analysis (e.g., PostGIS), and more. For non-professional users seeking to explore vector databases, they can readily download the open-source PostgreSQL or utilize managed services like Supabase and Neon to establish their own basic AI applications. Other than PostgreSQL, several open-source databases, including OpenSearch, ClickHouse, and Cassandra, have implemented their own vector search functionality. You do not need to adopt a new vector database if you have already used these systems.

https://gh-api.clickhouse.tech/play?user=play#U0VMRUNUICogRl...

I subscribe to this issue to keep up with updates:

https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

And ofc, for those that don't know, the official API https://github.com/HackerNews/API