Apache Pulsar VS studio

Stream-processing platforms such as Apache Kafka, Apache Pulsar, or Redpanda are specifically engineered to foster event-driven communication in a distributed system and they can be a great choice for developing loosely coupled applications. Stream processing platforms analyze data in motion, offering near-zero latency advantages. For example, consider an alert system for monitoring factory equipment. If a machine's temperature exceeds a certain threshold, a streaming platform can instantly trigger an alert and engineers do timely maintenance.

Edit: Here's a good example of what I'm looking for: https://github.com/apache/pulsar. It is a full application that happens to be deployed (or deployable) with Terraform, and the configuration files are available.

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

* Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

* The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

-- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

-- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

-- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

* WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

* BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

In recent years, it has become apparent that almost no production system is complete without real-time data. This can also be observed through the rise of streaming platforms such as Apache Kafka, Apache Pulsar, Redpanda, and RabbitMQ.

It's colored black on the refreshed Apache Pulsar site. https://pulsar.apache.org/

So many tools out there, its just which one do you like, I guess. I like Kafka. Works for our environment and we have a few clusters. People have brought up Cribl to replace our kafka (havent really looked into Cribl and we also run NiFi). I have even heard https://pulsar.apache.org/ , which seems to be almost another flavor of Kafka.

RisingWave is an open-source streaming database that has built-in fully-managed CDC source connectors for various databases, also it can collect data from other sources such Kafka, Pulsar, Kinesis, or Redpanda and it allows you to query real-time streams using SQL. You can get a materialized view that is always up-to-date.

> Especially a tool like RViz is always missing. And in many many robotics video I see (of a moderately complex robot), there's ROS's RViz on some screen.

I would love the future robotics development stack to be more modular, so that (for example) future middleware solutions don't need to also bundle their own visualization software. This was direct inspiration for creating Foxglove Studio[0] for visualization and MCAP[1] for logging - both work great with ROS, or equally well without it.

[0] https://github.com/foxglove/studio

[1] https://github.com/foxglove/mcap

That's the true strength of combining existing open source solutions. Thanks for the great support of Foxglove Inc. over the past few years.

Foxglove | Remote (US time zones) | Full Time or Contract | https://foxglove.dev/

Foxglove is the leading observability platform for robotics developers. We help robotics and AV companies log, ingest, organize, and visualize multimodal data. Join our small, highly experienced, and fast moving team. Prior experience working in robotics or AV is a bonus but not necessary.

* Senior Frontend Engineer (TypeScript, React, bonus if you have WebGL experience)

* Senior Full Stack Engineer (TypeScript, Go)

* Senior SRE/Infrastructure/DevOps Engineer (GCP, AWS, Azure, Terraform, K8s)

https://foxglove.dev/careers

Email in profile if you have questions.

It's a common misconception that you need a ML or robotics PhD to work in the field. There are plenty of frontend & backend web engineering, data engineering, and infrastructure/devops roles available at robotics companies - you don't need to be a domain expert.

For example, at Foxglove[0] we are building open source web-based visualization and data management for robotics (shameless plug: currently hiring).

[0] https://foxglove.dev/

You can use the rosbag commandline-tool to figure out all of this: http://wiki.ros.org/rosbag/Commandline rosbag info for example can give you some of the type/number of messages information. For more complex stuff, you could use https://foxglove.dev/ to visualize and inspect rosbags

Web-based data visualization for robotics and self-driving. Robotics is such an interesting industry, and we're only scratching the surface of what new tools are needed.

Try it live here (hit "view sample data"): https://studio.foxglove.dev/

And it's open source! https://github.com/foxglove/studio

Shameless plug - we're hiring: https://foxglove.dev/careers

When our team at Foxglove announced MCAP, we wanted to empower teams to spend less time building commodity tools and more time tackling their hardest robotics challenges.

Foxglove | Remote (Americas/Oceania) | Full Time or Contract | https://foxglove.dev/

Dev tools for robotics and autonomous vehicles.

Robotics will have a massive positive impact on the world economy and global human productivity over the coming decade. At Foxglove, we're building powerful web-based open source & commercial tools to accelerate this trend.

We're a small, highly experienced, and fast moving team. We're looking to hire several more people (most roles don't require previous robotics experience, unless specified below):

- Senior Frontend Engineer (react, typescript, some webgl+wasm)

- Senior Full Stack Engineer (typescript, some golang)

- First Product Manager (robotics/AV experience required)

- First Account Executive (infrastructure SaaS experience required)

https://foxglove.dev/

https://foxglove.dev/careers

React + Typescript. Forgot to mention it’s open source so you can see for yourself:

https://github.com/foxglove/studio