faust VS materialize

We have many similar ideas and paradigms, as one would expect from modern frameworks tackling the same problem. However, Faust has been deprecated and no longer managed or supported (there is a fork being maintained by the community at https://github.com/faust-streaming/faust). I guess Robinhood at some point decided not to spend any more resources on it, an often destiny of such side projects by large corporations. On the other hand, we hope to stay around for a long time and build to a framework and a community that loves it. This initial version was built using many libraries in order to have a working system and to gather feedback from the large community. We plan to replace eventually all, or at least most of dependancies, and replace then with Rust lib with Python bindings.

Just wanted to add that there is an actively maintained fork called faust-streaming, you can find it here: https://github.com/faust-streaming/faust

That's the basics yes. You have a pletora of things coming next. One is "Windowing" mentioned in the article, it's well explained and maybe it looks simple, but when you start with it, takes some time to wrap your mind around it.

The other things in kafka world are stateful transformations, which you would normally do using Java's Flink. The closest in python is Faust (the fork) [0]. What are stateful aggregations? something like doing SQL on top of a topic: group_by, count, reduce, and joins. So similar to SQL that you have kSQL [1].

Consumer groups IMO falls under basic usage, if you need to scale, take a look at it, and what are partitions and replicas, with that in mind, you'll be ok.

[0]: https://github.com/faust-streaming/faust

[1]: https://www.confluent.io/blog/ksql-streaming-sql-for-apache-...

The forked one, https://github.com/faust-streaming/faust, has been updated but still doesn’t seem to support joins.

Faust: Python framework

I’d like to use Python for this so I came across Faust but I’m not sure if it’s possible to create HTTP requests through Faust or if there are better alternatives.

There is a community fork which was stale for a while but got a new commit a couple days ago, this one might be usable, but is still quite risky (70 open issues etc.): https://github.com/faust-streaming/faust

There is a community which forked and actively develops faust, here’s the https://github.com/faust-streaming/faust

Hi, thanks for the answer, I'm actually using virtualenv with the requirements of the file associated with the example. However I wanted to tell you that due to covid the project is suspended but only temporarily, but there is an active fork https://github.com/faust-streaming/faust. But the example in the fork doesn't work either.

[2] https://materialize.com/

To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize, DeltaStream, and TimePlus. While they each have distinct commercial and technical approaches, their overarching goal remains consistent: to offer users cloud-based streaming database services.

> Materialize no longer provide the latest code as an open-source software that you can download and try. It turned from a single binary design to cloud-only micro-service

Materialize CTO here. Just wanted to clarify that Materialize has always been source available, not OSS. Since our initial release in 2020, we've been licensed under the Business Source License (BSL), like MariaDB and CockroachDB. Under the BSL, each release does eventually transition to Apache 2.0, four years after its initial release.

Our core codebase is absolutely still publicly available on GitHub [0], and our developer guide for building and running Materialize on your own machine is still public [1].

It is true that we substantially rearchitected Materialize in 2022 to be more "cloud-native". Our new cloud offering offers horizontal scalability and fault tolerance—our two most requested features in the single-binary days. I wouldn't call the new architecture a microservices design though! There are only 2-3 services, each quite substantial, in the new architecture (loosely: a compute service, an orchestration service, and, soon, a load balancing service).

We do push folks to sign up for a free trial of our hosted cloud offering [2] these days, rather than trying to start off by running things locally, as we generally want folks' first impression of Materialize to be of the version that we support for production use cases. A all-in-one single machine Docker image does still exist, if you know where to look, but it's very much use-at-your-own-risk, and we don't recommend using it for anything serious, but it's there to support e.g. academic work that wants to evaluate Materialize's capabilities to incrementally maintain recursive SQL queries.

If folks have questions about Materialize, we've got a lively community Slack [3] where you can connect directly with our product and engineering teams.

[0]: https://github.com/MaterializeInc/materialize/tree/main

Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.

https://github.com/timelydataflow/differential-dataflow

https://materialize.com/

Materialize | Full-Time | NYC Office or Remote | https://materialize.com

Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.

Materialize is the operational data warehouse built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.

Senior/Staff Product Manager - https://grnh.se/69754ebf4us

Senior Frontend Engineer - https://grnh.se/7010bdb64us

===

Investors include Redpoint, Lightspeed and Kleiner Perkins.

Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/

You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.

That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.

Engineering Manager, Compute - https://grnh.se/4e14099f4us

Senior Product Manager - https://grnh.se/587c36804us

VP of Marketing - https://grnh.se/9caac4b04us

How does it compare to https://materialize.com/ ?