Apache Heron: A realtime, distributed, fault-tolerant stream processing engine

• streamparse

  1 1,490 2.7 Python

  Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.

  

Wonder why this is getting posted today in particular?

The quick summary here is that this was a clean-house rewrite of Apache Storm done by an internal team at Twitter. As an open source project history refresher, Apache Storm was originally built by a startup called Backtype, and the project was led by Nathan Marz, the technical founder of Backtype. Then, Backtype was acquired by Twitter, and Storm became a major component for large-scale stream processing (of tweets, tweet analytics, and other things) at Twitter.

I wrote a summary of the "interesting bits" of Apache Storm here:

https://blog.parse.ly/storm/

However, at a certain point, Nathan Marz left Twitter, and a different group of engineers tried to rethink Storm inside Twitter. There was also a lot of work going on around Apache Mesos at the time. Heron is kind of a merger of their "rethinking" of Storm while also making it possible to manage Storm-like Heron clusters using Mesos.

But, I don't think Heron really took off. Meanwhile, Storm got very, very stable in the 1.x series, and then had a clean-house rewrite from Clojure to Java in the 2.x series. The last stable/major Storm release was in 2020.

Storm provides a stream processing programming API, a multi-lang wire protocol, and a cluster management approach. But certain cluster computing problems can probably be better solved at the infrastructure layer today. That said, it's still a very powerful system; on my team, we process 75K+ events per second across hundreds of vCPU cores and thousands of Python processes by combining Storm and Kafka with our open source project, streamparse.

https://github.com/Parsely/streamparse

(Also, I'd be remiss if I didn't mention -- if you're interested in stream processing and distributed computing, we are hiring Python Data Engineers to work on a stack involving Storm, Spark, Kafka, Cassandra, etc.) -- https://www.parse.ly/careers/python_data_engineer

• thanos

  66 12,577 9.6 Go

  Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

  

> Sometimes, I feel like Apache foundation is like Thanos

Soon: https://thanos.io/

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project