readyset VS googleapis

- Some platforms like Supabase Realtime [3] and Firebase offer subscription models to database changes, but these solutions fall short when dealing with complex queries involving joins or group-bys.

My vision is that the modern frontend to behave like a series of materialized views that dynamically update as the underlying data changes. Current state management libraries handle state trees well but don't seamlessly integrate with relational or graph-like database structures.

The only thing I can think of is to implement it by myself, which sounds like a big PITA.

Anything goes, Brainstorm with me. Is it causing you headaches as well? Are you familiar with an efficient solution? how are you all tackling it?

[1] https://readyset.io/

Postgresql + MySQL Cache https://github.com/readysettech/readyset

I just wanted to give a high five for having Jepsen tests for this: https://github.com/readysettech/readyset/tree/stable-240117/...

This example is a great use case for partial incremental view maintenance systems like ReadySet: you automatically get something like the “prepopulating the cache” section (toward the end of the blog) while only caching the data the application is using, and avoiding the need to manually implement any sort of invalidation logic.

(Disclaimer: I used to work for them, but don’t anymore. It’s all available for free on GitHub though for anyone interested: https://github.com/readysettech/readyset)

There are systems that will do that for you like https://readyset.io/.

ReadySet

readyset.io is the company that jonhoo was associated with for work on noria

Might be interesting to try out https://readyset.io for this use case.

On the caching topic, I found the ReadySet(né Noria) approach to be extremely interesting.

Rich Error Model: This model enables servers to return and clients to consume additional error details expressed as one or more protobuf messages. It further specifies a standard set of error message types to cover the most common error (QuotaFailure, PreconditionFailure, BadRequest, etc). When an error occurs, the server returns the appropriate status code along with an optional error message.

They list search in their public api

https://github.com/googleapis/googleapis/blob/288aa7fb71c9b6...

We structure the repo according to proto packages. It's quite similar to how the googleapis repository is structured.

It's unfortunate that the spec doesn't contain custom fields to a sub-object like other RPC specs, like proto Status [1]. They should have had the message go into a field named "message" and not "detail". And have a field like "details" where the opaque type is serialized, which should be named by the "type" field. The problem is that systems with existing error types may have field name conflicts with type, title, status, detail, or instance, so we'd just dump the actual error into a custom "extension member" which by definition, isn't standard.

[1] https://github.com/googleapis/googleapis/blob/1c8a25ab153eef...

Protobuf (https://github.com/googleapis/googleapis)

All the required changes can be viewed in our last demo, the grpc-rest-app implementation. First, we need to update our proto service interface to help the proxy service make our gRPC service methods available at the right URLs and for the correct HTTP operations. To do this, the Google API HTTP library provides annotations we can add to our proto to describe the correct mappings. The buf tool allows us to include the googleapis dependency as a plugin in our buf.yaml file).

I think this only makes sense if the 3rd party is also throwing custom exceptions.

If you want to reduce coupling you should avoid throwing custom exceptions at all. Semantic information can go in the error message and log. The error type should be used to indicate to your program whether an error is recoverable, retriable or some other action needs to be taken. For example google on has 16 canonical error codes for all APIs.

https://github.com/googleapis/googleapis/blob/master/google/...

OpenAPI and possibly developing reusable, versioned client libraries could help, but it's a major undertaking that gRPC makes redundant. I'd be tempted to use grpc-gateway even if I had to implement a REST API. Try looking into buf and monorepo structures for proto management, e.g. something like GoogleCloudPlatform/microservices-demo. For more thorough proto and grpc-gateway definition examples, see googleapis/googleapis.

Firestone does provide global consistency, so the following quote is incorrect:

> In Cloud Firestore, the data on the client are loaded from the database at different points in time. Even if you listen for realtime updates, results from separate queries will not remain in sync. This creates consistency anomalies and bugs in your app.

Here is a link to the protocol documentation that the clients use to support it: https://github.com/googleapis/googleapis/blob/d0b394f188e8c3...

I'd link to the client implementation but it's quite involved.

Well there is Timestamp defined as a well known type which is available to all implementations despite not being a primitive type [1]. Plus one is obviously able to define any other custom types if necessary- eg as seen in [2].

[1] https://developers.google.com/protocol-buffers/docs/referenc...

[2] https://github.com/googleapis/googleapis/blob/master/google/...