Kafka Is Dead, Long Live Kafka

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • redpanda

    Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

    that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes?

    it's a misconception that redpanda is simply a better kafka. the way to think about it is that is a new storage engine, from scratch, that speaks the kafka protocol. similar to all of the pgsql companies in a different space, i.e.: big table pgsql support is not a better postgres, fundamentally different tech. you can read the src and design here: https://github.com/redpanda-data/redpanda. or an electric car is not the same as a combustion engine, but only similar in that they are cars that take you from point a to point b.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • tiered-storage-for-apache-kafka

    RemoteStorageManager for Apache Kafka® Tiered Storage

    I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

    * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

    * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

    -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

    -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

    -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

    -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

    -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

    * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

    * BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

    I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

  • I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

    * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

    * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

    -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

    -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

    -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

    -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

    -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

    * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

    * BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

    I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

  • Apache Pulsar

    Apache Pulsar - distributed pub-sub messaging system

    I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

    * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

    * The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

    -- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

    -- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

    -- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

    -- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

    -- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

    * WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

    * BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

    I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

  • transit

    A bytes first implementation of the Kafka API within an S3 keyspace

    This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit.

    FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies.

    I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Querying microservices in real-time with materialized views

    4 projects | dev.to | 30 Apr 2023
  • How Streaming database differs from a Traditional database?

    3 projects | dev.to | 31 Mar 2023
  • Show HN: Franz – A desktop client for Apache Kafka

    3 projects | news.ycombinator.com | 5 Feb 2024
  • Help finding open source Terraform configurations that are not educational projects or developer tools

    2 projects | /r/Terraform | 28 Sep 2023
  • Analyzing Real-Time Movie Reviews With Redpanda and Memgraph

    2 projects | dev.to | 6 Jul 2023