foundationdb VS winget-pkgs

Actually, Apple does this for iCloud! They use FoundationDB[1] to store billions of databases, one for each user (plus shared or global databases).

See: https://read.engineerscodex.com/p/how-apple-built-icloud-to-...

Discussed on HN at the time: https://news.ycombinator.com/item?id=39028672

[1]: https://github.com/apple/foundationdb https://en.wikipedia.org/wiki/FoundationDB

Is this the language in question? https://github.com/apple/foundationdb/blob/main/flow/README....

> I wondered if perhaps this WASM use case for a cross-language library was already just as possible and ergonomic using language bindings and maybe thats why this use case doesnt seem like a big deal to people.

Yeah that’s the reason. You don’t notice it a lot of the time, but FFIs are everywhere already. The most common foreign function interface is basically the ability to call C code, or have functions made available to C code. C is used because everyone knows it and it’s simple. And most languages either compile to native code (eg rust) - which makes linking to C code easy. Or the runtime is implemented in C or C++ (eg V8, Ruby). In languages like that, the standard library is already basically implemented via a FFI to C/C++ code.

I’ve got an iOS app I’m working on that’s half rust and half swift, with a touch of C in the middle. The bindings work great - the whole thing links together into one binary, even with link time optimizations. But the glue code is gross, and when I want to fiddle with the rust to Swift API I need to change my code in about 4 different places.

Most FFIs are a one to many relationship in that if you write a clean C API, you can probably write bindings in every language. But you don’t actually want to call naked C code from Ruby or Javascript. Good bindings will make you forget everything is done via ffi. Eg numpy. I haven’t looked at the wasm component model proposal - I assume it’s trying to make this process cleaner, which sounds lovely.

I maintain the nodejs bindings for foundationdb. Foundationdb bindings are all done via ffi linking to their C code. And the API is complex, using promises and things. I find it really interesting browsing their official bindings to go, Java, Python and Ruby. Same bindings. Same wrapped api. Same team of authors. Just different languages. And that’s enough to make the wrapper wildly different in every language. From memory the Java ffi wrapper is 4x as much code as it is in Ruby.

https://github.com/apple/foundationdb/tree/main/bindings

Isn’t apple server on C/C++?

They have their own db in that

https://github.com/apple/foundationdb

> Any meaningful technology is open source.

Clearly untrue, however FoundationDB is open source, with a permissive license.

https://github.com/apple/foundationdb

So is much of the operational tooling for it:

https://github.com/FoundationDB/fdb-kubernetes-operator

I don't recall any of those details but the test involved injecting a bogus block device that always returns garbage, and noting that this results in garbage records returned from client queries. And I don't think those kinds of issues have been eradicated, browsing through their github issues there are people trying to recover corrupted clusters. https://github.com/apple/foundationdb/issues/2480

For my current job, I have to install FoundationDB (they have a github repo) as a dependency for the project's code.

To clarify, FoundationDB has some documented multi-region features, but it's not at all clear that anybody who runs it at scale relies on those multi-region features [1]. Even if they do, it's not obvious they run this Fearless DR mode at any appreciable latency.

Geo-replication is an area where Anna really shines, and foundation pays severe costs. All of the concurrency control problems have contention footprints on the order of round-trips. Fully optimistic concurrency control needs to expose backoffs quite a few round-trips to be live. Even pessimistic concurrency control requires some number of round trips, probably at least 1.5 optimally, but in practice in most tuned systems, probably 3. A heck of a lot of use cases make sense at 0 global RTTs and don't at 1+. The ability to tell the database how to manage concurrency, and then providing causal is ultimately the best you can do. That's awesome.

At the end of the day, I have to believe that there's a whole big mess of applications we'll build this, on systems in the portion of the design space Anna is choosing, one day. This is only recently not cutting edge research, but it's definitely still research. We don't know how to model these concepts at a high-level and in a composable way where the masses of software developers can engage with them.

It's interesting to think about how long ago we were graced with BAYOU [2], that thing was ahead of its time. I suspect it's going to take a little while longer before these sorts of techniques make their way into data storage primitives we think of as part of the standard vernacular, but I believe we'll get there eventually.

Databases are fun!

[1] https://github.com/apple/foundationdb/wiki/Multi-Region-Repl...

7.0 is now available: https://github.com/microsoft/winget-pkgs/pull/147886

I researched this for a WinGet thing: https://github.com/microsoft/winget-pkgs/pull/110618

Ah, reminds me of that time I requested a .editorconfig file in a Microsoft repo: https://github.com/microsoft/winget-pkgs/issues/329

Take dropbox as an example. This is what the yaml manifest looks like for that if you install it through winget. It literally has a hardcoded link to an .exe installer hosted by dropbox and then just set the flags to silent. I am not spreading misinformation, you are.

It's not quite the same though, as there are different considerations when using a repository of things a unified group has decided should be included and built (or slightly modified existing) packages for and a repo where anyone can submit a package that will go through some level of vetting. In the end I still believe most this discussion is really about individuals and how much trust they apply towards different groups and sources and is not really about Linux or Windows in particular as much.

1: https://github.com/microsoft/winget-pkgs

I never used winget, but probably: - https://github.com/microsoft/winget-pkgs/issues/107858 - https://github.com/Genymobile/scrcpy/issues/4027

It's probably not on the Store, winget pulls from both the Store and a community collection of manifests on GitHub: https://github.com/microsoft/winget-pkgs

I think that's part of the problem, if you don't have that package manager to bootstrap your signature key ring, DNS is your next best bootstrap. It is, of course, a terrible bootstrap for trust, but it is one so many users on Windows have been relying on for such a long time.

For power users on any modern Windows 10/Windows 11 there is at least WinGet now. Its manifests repo is becoming a very interesting (open) source of truth for common Windows applications. Admittedly, it in most cases doesn't seem to be checking specific code signatures in most cases either, but at least includes SHA checksums.

For instance, 7zip's manifests: https://github.com/microsoft/winget-pkgs/tree/master/manifes...

It's too bad there's still not a great option for "average user that doesn't know/trust how to use a CLI", given how sadly polluted the Microsoft Store can be for many common, especially Open Source, applications. For direct instance, because winget kindly includes Microsoft Store results when searching, there is a "7zip 22" in the Microsoft Store that costs some amount of money (winget details say "PaidUnknownPrice" for the pricing information; I'm on a corporate machine right now with the actual Store access locked so can't search in the actual Store right now) and the Publisher is listed as RepackagerExpress.com. (That website currently doesn't go anywhere, giving it a spot check.)

Having seen this, I may boot up my personal machine and try to report this specific Store listing for violating the Store's Open Source policies, though I'm unsure if such whackamole is all that useful. (Seems like it might be a useful winget feature request for it to provide Store Report URLs.)

For example, see that Firefox has /S here.