The Fastest Way to Build the Fastest Data Products. Build data-intensive applications and services in SQL — without pipelines or caches — using materialized views that are always up-to-date. (by MaterializeInc)
Yep, cache invalidation is hard because you can't just cache /users/1/networkstatus as a function of the request itself; its underlying value is a complex function of values fetched from possibly dozens of tables and services, any of which could change at any time without any immediately discernable connection to User 1.
OP's response to this, in another comment, is: "Now going back to your example with complicated dependencies. Maybe TTL is a better solution. With many dependencies, any changes from the dependency list might trigger invalidation. At some point, just doing TTL, would be simpler."
But what if you actually do want your dependency list to trigger invalidation?
https://materialize.com/ is the closest thing I know of to a "solution" for this - it lets you build nested, realtime materialized views on top of streaming and non-streaming data, all with SQL syntax and Postgres wire compatibility, that fully recalculate and recache their results whenever any input changes, no matter how deep (based on https://timelydataflow.github.io/timely-dataflow/ from Microsoft Research).
You could either use the outputs of such a system as a cache, or weave together a cache invalidation signaling system that sends a stream of keys that need to be evicted from cache at any time, directly into a Kafka topic. And then, of course, this could plug into Meta's cache invalidation consensus system. But it's remarkably disingenuous for Meta to pretend that the dependency side of this problem is trivial or can be solved by TTLs alone.
Why does dbt have so much hype/ metions in this subreddit?
1 project | reddit.com/r/dataengineering | 17 Jun 2022
Materialize - The fastest way to build the fastest data products
1 project | reddit.com/r/github_trends | 30 May 2022
cargo-machete: Remove unused dependencies with this one weird trick!
1 project | reddit.com/r/rust | 27 Apr 2022
2 projects | news.ycombinator.com | 5 Apr 2022
DATA PLATFORMS VS DATA WAREHOUSES VS DATA VIRTUALIZATION AND FEDERATED SQL ENGINE
1 project | dev.to | 30 Mar 2022