timely-dataflow VS Hasura

They have a bit about their technical foundation here[0].

Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research.

So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow.

[0]: https://docs.readyset.io/concepts/streaming-dataflow

[1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf

[2]: https://dl.acm.org/doi/10.1145/2517349.2522738

[3]: https://github.com/TimelyDataflow/timely-dataflow

And systems like timely dataflow, https://github.com/TimelyDataflow/timely-dataflow

Project looks cool! Glad you open sourced it. It could use some comments in the code base to help contributors ;). I also like the datafusion usage, that is awesome. BTW I work on github.com/bytewax/bytewax, which is based on https://github.com/TimelyDataflow/timely-dataflow another Rust dataflow computation engine.

Just found this https://github.com/TimelyDataflow/timely-dataflow and my heart skipped a beat.

Bytewax is a Python native binding to the Timely Dataflow library (written in Rust) for building highly scalable streaming (and batch) processing pipelines.

I am working on Bytewax, which is a Python stream processing framework built on Timely Dataflow. It is not exactly a Kafka integration framework because it is a more of a general stream processing framework, but might be interesting for you. We are focused on enabling people to more easily debug, containerize, parallelize and customize and less on enabling a declarative integration framework. It is still early days for us! And we are looking for feedback and ideas from the community.

We used to use a library called Pond.js, https://github.com/esnet/pond, but the reliance on Immutable.JS caused some performance pitfalls, so we wrote a system from scratch that deals with data in a batched streaming fashion. A lot of the concepts were borrowed from a Rust library called timely-dataflow, https://github.com/TimelyDataflow/timely-dataflow.

Though the name "Dataflow" might be an unfortunate name conflict with another Rust project: https://github.com/TimelyDataflow/timely-dataflow

> In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.

I see that now -- makes sense!

> Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.

I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.

[1] https://github.com/TimelyDataflow/timely-dataflow

Under the hood, Materialize uses Timely Dataflow (TDF) as the stream-processing engine. This allows Materialize to take advantage of the distributed data-parallel compute engine. The great thing about using TDF is that it has been in open source development since 2014 and has since been battle-tested in production at large Fortune 1000-scale companies.

> 2. ORMs do not hide SQL nastiness.

This is certainly true!

I mean: ORMs are now well known to "make the easy queries slightly more easy, while making intermediate queries really hard and complex queries impossible".

I think the are of ORMs is over. It simply did not deliver.

If a book on SQL is --say-- 100 pages, a book on Hibernate is 400 pages. So much to learn just to make the easy queries slightly easier to type? Just not worth it.

I prefer jooq any day over ORMs. And dont get me started over what tools like Hasuna have to offer.

There are also some languages (forgot the names) that are SQL-done-right. Select in the back, more type safe, more logic, more in the same steps as the query gets executed. These need to be adopted by PG and MySQL and we're good to go. (IMHO)

https://www.jooq.org/

https://hasura.io/

[4] https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md

Another strategy is to model access control declaratively and enforce it in the application layer. ZenStack (built above Prisma ORM) and Hasura are good examples of this approach. The following code shows how access policies are defined with ZenStack and how a secured CRUD API can be derived automatically.

Today, this ecosystem is going strong with new providers like Hasura, AppWrite and Supabase powering millions of projects. There are a few reasons people choose this style of hosting, especially if they are more comfortable with frontend development. BaaS lets them set up a database in a secure way, expose some business logic on top of the data, and connect via a dev-friendly SDK from their app or website code to save data easily. These modern tools build a blend of managed database with curated plugins such as authentication, great admin dashboards, and function as a service type capability - all in one package, and often offered as a integrated hosted service.

Hi! If you’ve ever thought about something like using GraphQL for something like this.. You might like Hasura. (Obligatory I work for Hasura)

We’ve got an OpenAPI import and you can setup cron-jobs or one-off jobs and do things like load in headers from the environment variables to pass through. There isn’t currently an easy journey for chaining multiple calls together without writing any code at all, but you can wrap pretty much any API endpoint via OpenAPI import or a custom action, and you can even make minor edits to things like the API contract format to change aliases/naming.

Our goal is to join all the things, databases and API’s. Most people know us for instant GraphQL API’s that give you CRUD on your database, but we also wrap APIs.

Not sure if something like this would fit your use-case and do check out some of the other things mentioned, but depending what you are trying to do I think Hasura might potentially work.

You can find out more here: https://hasura.io

Nested Mutation - 113 thumbs up, and still open since 2019... another case of not listening to the users?

Hasura is an open-source real-time GraphQL API server with a strong authorization layer on your database. You can subscribe to database events via webhooks. It can combine multiple API servers into one unified graphQL API. Hasura is a great tool to build any CRUD GraphQL API. Hasura does not have any authentication mechanisms; e.g., you need an auth server to handle sign-up and sign-in.