stargate VS Apache Pulsar

They suggested https://stargate.io

- https://github.com/stargate/stargate

Datastax Astra is built on Apache Cassandra. In addition to great documentation, Astra offers a robust free tier that can run small production workloads, pet projects, or just let you play—all for free, no credit card required. Cassandra can be tricky for hardcore SQL developers, because it uses a different slightly different query language (CQL), but when you get Astra, Stargate is there to let you interact with your data through APIs. Our open source Stargate product provides REST, GraphQL, and schemaless document APIs in addition to native language drivers. If you like them but don’t want to use our products, that’s fine! It’s completely open source and you can implement it on your own system.

DataStax built Stargate into Astra to give us, app developers, a natural data API stack which meshes with the Jamstack (or serverless stack of your choice). Stargate in Astra is built on the rock solid NoSQL data engine (Apache Cassandra) which powers Netflix, Instagram, Yelp, iCloud and other apps we all use everyday.

To make it easy for your app to interact with the database, we created Stargate.io. It’s an open-source data gateway with three APIs that work with Astra DB right out of the box. Instead of having to read up on different APIs and databases, all you have to do is pick one of the three Stargate APIs and get to work on your application.

Twelve-plus years after its invention, Cassandra is now used by approximately 90 percent of the Fortune 100, and it’s appeal is broadening quickly, driven by a rush to harness today’s “data deluge” with apps that are globally distributed and always-on. Add to this recent advances in the Cassandra ecosystem such as Stargate, K8ssandra, and cloud services like Astra DB, and the cost and complexity barriers to using Cassandra are fading into the past. So while it’s fair to say that while Cassandra might have been ahead of its time in 2007, it’s primed and ready for the data demands of the 2020s and beyond.

Until now, aggregate functions were only available using cqlsh (the CQL Shell). However, with the Stargate 1.0.25 release, they are now also available using the GraphQL API. In this blog entry, I’ll walk you through the process to get early access to this exciting new functionality in Stargate, and how to set up everything you need to test your own aggregate queries.

Stargate is an open-source data gateway that makes it simple to query any Cassandra database using GraphQL types, queries, and mutations. When you add the Stargate GraphQL API to a Cassandra deployment, it scans the database and automatically creates HTTP endpoints with GraphQL queries and mutations for the objects it finds.

BattleStax is implemented as a JAMStack app that uses Stargate, Netlify, DataStax Astra DB, and GitHub to demonstrate how to build and deploy an application using modern, scalable architectures. In this post, we’ll break down the video to help you quickly create your own BattleStax game using React and Redux — implemented with a CI/CD pipeline, global content delivery network (CDN), and Apache Cassandra®.

Stream-processing platforms such as Apache Kafka, Apache Pulsar, or Redpanda are specifically engineered to foster event-driven communication in a distributed system and they can be a great choice for developing loosely coupled applications. Stream processing platforms analyze data in motion, offering near-zero latency advantages. For example, consider an alert system for monitoring factory equipment. If a machine's temperature exceeds a certain threshold, a streaming platform can instantly trigger an alert and engineers do timely maintenance.

Edit: Here's a good example of what I'm looking for: https://github.com/apache/pulsar. It is a full application that happens to be deployed (or deployable) with Terraform, and the configuration files are available.

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

* Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

* The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

-- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

-- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

-- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

* WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

* BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

In recent years, it has become apparent that almost no production system is complete without real-time data. This can also be observed through the rise of streaming platforms such as Apache Kafka, Apache Pulsar, Redpanda, and RabbitMQ.

It's colored black on the refreshed Apache Pulsar site. https://pulsar.apache.org/

So many tools out there, its just which one do you like, I guess. I like Kafka. Works for our environment and we have a few clusters. People have brought up Cribl to replace our kafka (havent really looked into Cribl and we also run NiFi). I have even heard https://pulsar.apache.org/ , which seems to be almost another flavor of Kafka.

RisingWave is an open-source streaming database that has built-in fully-managed CDC source connectors for various databases, also it can collect data from other sources such Kafka, Pulsar, Kinesis, or Redpanda and it allows you to query real-time streams using SQL. You can get a materialized view that is always up-to-date.