Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
uwm-masters-thesis
My thesis for my Master's in Computer Science degree from the University of Wisconsin - Milwaukee.
-
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
wai
A language binding generator for `wai` (a precursor to WebAssembly interface types) (by wasmerio)
-
-
-
reference-crdts
Simple, tiny spec-compliant reference implementations of Yjs and Automerge's list types.
The simplest way to get involved is to start attending the biweekly standardization meetings. The agendas are organized here: https://github.com/WebAssembly/meetings
To attend the meetings, first join the W3C WebAssembly Community Group here: https://www.w3.org/groups/cg/webassembly, then email the CG chairs at [email protected] to ask for an invite.
From there you'll get a sense of who folks are so you can pair names with faces when contributing to the various proposal discussions on the many proposal repos listed here: https://github.com/webassembly/proposals.
To get a sense of how things are run and decided, read the process documents here: https://github.com/WebAssembly/meetings/tree/main/process. The TL;DR is that the community group and its subgroups decide everything by consensus via votes during the meetings.
Well, backwards compatibility is really a spectrum and the interpretation of the concept can fall at either end of the spectrum depending on what a person is working on. As far as wasm is concerned, there are at least two known discrepancies that made into this draft as documented here https://github.com/WebAssembly/spec/issues/1439.
Especially the last comment by Alex here seems interesting to me – on one hand the observable result of the execution changes for the same program. On another, who is going to bother writing such programs? Should WebAssembly specification be allowed to assume nobody does and make changes such as these?
Neat, a few of these accepted proposals were the topic of my Master's thesis a couple years ago: https://github.com/jacobmischka/uwm-masters-thesis
Exciting to see a major release, though admittedly it seems a little arbitrary if there aren't any breaking changes (and very problematic if there are?).
The simplest way to get involved is to start attending the biweekly standardization meetings. The agendas are organized here: https://github.com/WebAssembly/meetings
To attend the meetings, first join the W3C WebAssembly Community Group here: https://www.w3.org/groups/cg/webassembly, then email the CG chairs at [email protected] to ask for an invite.
From there you'll get a sense of who folks are so you can pair names with faces when contributing to the various proposal discussions on the many proposal repos listed here: https://github.com/webassembly/proposals.
To get a sense of how things are run and decided, read the process documents here: https://github.com/WebAssembly/meetings/tree/main/process. The TL;DR is that the community group and its subgroups decide everything by consensus via votes during the meetings.
You may want to look into WASM interface types, which is defining what amounts to am IDL for WASM and different languages have common calling conventions: https://hacks.mozilla.org/2019/08/webassembly-interface-type...
I don’t know if there’s a better intro article. I believe this is the current iteration of the proposal: https://github.com/WebAssembly/interface-types/blob/main/pro...
Are subprojects intended to eventually make it into the main spec or do they remain standalone? GC, threads, atomics, component-model, interface-types... Im not sure what to expect to eventually make it into this spec & what will continue developmemt semi-independently.
Personally https://github.com/WebAssembly/component-model/pull/23 feels like it marks the start of webassembly's first full era. We need interoperation, webassembly needs to really be embeddable, & only then can we really understand & explore what webassembly means to the world.
(Rust got an early start with wasm-bindgen, doing all the lifting on their side, but it's not reasonable & only semi-interoperable having each language rebuild that bindgen system & abingor itself... hence canonical abi & component-model.)
Also take a look at wit-bindgen [1], which is basically a prototype for interface types and supports multiple languages.
I agree we that interface types/components will be a game changer.
And also that progress on all the important proposals (GC, interface types, linking, stack control, threads) has been painfully slow.
[1] https://github.com/bytecodealliance/wit-bindgen
https://github.com/WebAssembly/proposals
https://github.com/WebAssembly/meetings/blob/main/process/ph...
Here's the proposal:
https://github.com/WebAssembly/gc
And you can see it's being actively worked on:
https://github.com/WebAssembly/gc/commits/main
wit-bingen is great. For those that want to use wit-bindgen with Wasmer (tl;dr I work at Wasmer) here's the repo:
https://github.com/wasmerio/wit-bindgen
Interestingly enough Dart is also pretty deep into adding WASM as a compile target to act as an alternative to the current compile to optimised JS approach they do currently in the browser.
Work going on here but not much to look at now just source and no docs https://github.com/dart-lang/sdk/tree/main/pkg/dart2wasm
> In this case, the bottleneck at 9 million LoC is not CPU cycles but memory usage. That's where I am considering pushing down into WebAssembly
How often does this come up in practice? I can't think of many files I've opened which were 9 million lines long. And you say "LoC" (lines of code). Are you doing syntax highlighting on 9 million lines of source code in javascript? Thats impressive!
> I guess my point is why do you need balanced trees? Is this a CRDT specific thing? Can you implement CRDT with just an array of lines / gap buffer?
Of course! Its just going to be slower. I made a simple reference implementation of Yjs, Automerge and Sync9's list types in javascript here[1]. This code is not optimized, and it takes 30 seconds to process an editing trace that diamond types (in native rust) takes 0.01 seconds to process. We could speed that up - yjs does the same thing in 1 second. But I don't think javascript will ever run as fast as optimized rust code.
The b-tree in diamond types is used for merging. If you're merging 2 branches, we need to map insert locations from the incoming branch into positions in the target (merged) branch. As items are inserted, the mapping changes dynamically. The benchmark I've been using for this is how long it takes to replay (and re-merge) all the changes in the most edited file in the nodejs git repository. That file has just shy of 1M single character insert / delete operations. If you're curious, the causal graph of changes looks like this[2].
Currently it takes 250ms to re-merge the entire causal graph. This is much slower than I'd like, but we can cache the merged positions in about 4kb on disk or something so we only need to do it once. I also want to replace the b-tree with a skip list. I think that'll make the code faster and smaller.
A gap buffer in javascript might work ok... if you're keen, I'd love to see that benchmark. The code to port is here: [3]
> Undo support -> In which case, you only have to stack / remember the set of commands and not have to store the state on every change. I'm not sure if this overlaps with the data structure choice, other than implementation details.
Yeah, I basically never store a snapshot of the state. Not on every change. Not really at all. Everything involves sending around patches. But you can't just roll back the changes when you undo.
Eg: I type "aaa" at position 0 (the start of the document). You type "bbb" at the start of the document. The document is now "bbbaaa". I hit undo. What should happen? Surely, we delete the "aaa" - now at position 3.
Translating from position 0 to position 3 is essentially the same algorithm we need to run in order to merge.
> I was just looking into TypedArrays.
I tried optimizing a physics library a few years ago by putting everything in typedarrays and it was weirdly slower than using raw javascript arrays. I have no idea why - but maybe thats fixed now.
TypedArrays are useful, but they're no panacea. You could probably write a custom b-tree on top of a typedarray in javascript if you really want to - assuming your data also fits into typedarrays. But at that point you may as well just use wasm. It'll be way faster and more ergonomic.
[1] https://github.com/josephg/reference-crdts
[2] https://home.seph.codes/public/node_graph.svg
[3] https://github.com/josephg/diamond-types/tree/master/src/lis...
> In this case, the bottleneck at 9 million LoC is not CPU cycles but memory usage. That's where I am considering pushing down into WebAssembly
How often does this come up in practice? I can't think of many files I've opened which were 9 million lines long. And you say "LoC" (lines of code). Are you doing syntax highlighting on 9 million lines of source code in javascript? Thats impressive!
> I guess my point is why do you need balanced trees? Is this a CRDT specific thing? Can you implement CRDT with just an array of lines / gap buffer?
Of course! Its just going to be slower. I made a simple reference implementation of Yjs, Automerge and Sync9's list types in javascript here[1]. This code is not optimized, and it takes 30 seconds to process an editing trace that diamond types (in native rust) takes 0.01 seconds to process. We could speed that up - yjs does the same thing in 1 second. But I don't think javascript will ever run as fast as optimized rust code.
The b-tree in diamond types is used for merging. If you're merging 2 branches, we need to map insert locations from the incoming branch into positions in the target (merged) branch. As items are inserted, the mapping changes dynamically. The benchmark I've been using for this is how long it takes to replay (and re-merge) all the changes in the most edited file in the nodejs git repository. That file has just shy of 1M single character insert / delete operations. If you're curious, the causal graph of changes looks like this[2].
Currently it takes 250ms to re-merge the entire causal graph. This is much slower than I'd like, but we can cache the merged positions in about 4kb on disk or something so we only need to do it once. I also want to replace the b-tree with a skip list. I think that'll make the code faster and smaller.
A gap buffer in javascript might work ok... if you're keen, I'd love to see that benchmark. The code to port is here: [3]
> Undo support -> In which case, you only have to stack / remember the set of commands and not have to store the state on every change. I'm not sure if this overlaps with the data structure choice, other than implementation details.
Yeah, I basically never store a snapshot of the state. Not on every change. Not really at all. Everything involves sending around patches. But you can't just roll back the changes when you undo.
Eg: I type "aaa" at position 0 (the start of the document). You type "bbb" at the start of the document. The document is now "bbbaaa". I hit undo. What should happen? Surely, we delete the "aaa" - now at position 3.
Translating from position 0 to position 3 is essentially the same algorithm we need to run in order to merge.
> I was just looking into TypedArrays.
I tried optimizing a physics library a few years ago by putting everything in typedarrays and it was weirdly slower than using raw javascript arrays. I have no idea why - but maybe thats fixed now.
TypedArrays are useful, but they're no panacea. You could probably write a custom b-tree on top of a typedarray in javascript if you really want to - assuming your data also fits into typedarrays. But at that point you may as well just use wasm. It'll be way faster and more ergonomic.
[1] https://github.com/josephg/reference-crdts
[2] https://home.seph.codes/public/node_graph.svg
[3] https://github.com/josephg/diamond-types/tree/master/src/lis...