seshat VS matrix-spec-proposals

> Can you already say something about encrypted search support in Element X?

It's planned, and should be way easier to do than on old Element given we can hook the clientside search indexer (https://github.com/matrix-org/seshat), which happens to be in Rust, directly into matrix-rust-sdk and thence both EIX and EAX. Whereas previously we'd have had to do custom integs with Seshat on EI and EA like the current one on Element Desktop, which just becomes a maintenance nightmare. Hopefully should be ready in the next months.

> Also since you mentioned it, is multiaccount support something that we can realistically expect to see in Element X someday (maybe even in a v1.0?). If I remember correctly for the previous iterations of the clients the status was mostly "maybe one day" or "the architecture does not really support it".

Definitely. We actually had it pre-Element in Matrix Console, and then it got lost in Element. The hope is to get it into Element X relatively early.

https://github.com/matrix-org/seshat - we’ve had it for years on desktop; just need to hook it up on mobile. it’s currently blocked behind unifying sdks, so we don’t have to do it twice in iOS & Android.

Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.

The competitive gap with Discord in terms of media quality is probably something like:

* Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, and letting anyone bring their SFU to the party.

* Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/

* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.

* Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important.

We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.

We can probably also be more bullish about supporting new audio codecs like Lyra.

Messages are stored in encrypted form on the Beeper server and the Beeper client has a local search index (the same one used by Element desktop: https://github.com/matrix-org/seshat)

Not only are they actually very closely linked, in that Element operates matrix.org, but to a new user (told to try Matrix -- what is this Element thing?) there's no difference.

I onboarded a family member onto my Matrix server with FluffyChat as the client. This person is a power user, fairly technical, yet still refers to the chat as "FluffyChat" and although I've explained several times that choosing FluffyChat was maybe a mistake and they should use Element, it never seems to really click that multiple clients are possible.

And really, they aren't possible. They have different subsets of features.

If you want to see a trash can fire, just try to follow the discussion for adding custom emoji to Matrix: https://github.com/matrix-org/matrix-spec-proposals/pull/195...

it's been going on for years. It's a feature the competitors have had for half a decade, as long as this discussion has been ongoing. I've been watching this issue for half a decade thinking "surely they'll decide on something" but mostly all I've been convinced of is this: Matrix is design by committee in all of the worst aspects and at every level of design. If anything gets done at all, it's a convoluted mess, and it's a miracle that it even happens.

I wish community software developers would focus their attention.. somewhere else.

So Matrix also has account portability (almost) - https://github.com/matrix-org/matrix-spec-proposals/blob/keg... and https://github.com/devonh/matrix-spec-proposals/blob/cryptoI..., implemented in Dendrite. Unfortunately dev is paused on it currently thanks to lack of $ though.

The AP approach (prioritising portable identities over portable account data) is cute though, and perhaps we should have prioritised that as an alternative to fullblown cryptographic IDs & account portability.

Luckily, it doesn't matter what individuals expect. There is written documentation on what the foundation is supposed to do or not to do: https://github.com/matrix-org/matrix-spec-proposals/blob/mai...

Notably, "Code Core Team members must arrange their own funding for their time", which I understand as such that the Foundation does not pay directly the developers (same as other standards organizations like IETF).

Main tasks of Matrix.org Foundation is maintaining the spec, documentation, owning IP, promotion and the matrix.org home server. The home server is "generously hosted" by UpCloud (i.e. is not using New Vector EMS), at least according to the matrix.org website.

Looking again at MSC1779, I noticed it says that one function of The Matrix.org Foundation is "Owns the copyright of the reference implementations of Matrix (i.e. everything in https://github.com/matrix-org). By assigning copyright to the Foundation, it’s protected against New Vector ever being tempted to relicense it." That protection apparently wasn't very effective, but also notably, New Vector and their leadership clearly have shown to not stand behind the goals of the Foundation. As the leadership of New Vector is also part of the leadership of the Foundation, I see some huge potential for COI here.

The main remaining Nebuchadnezzar issue is mitigating server-controlled group membership. The first step has been to kill off the 1st gen E2EE implementations, which were responsible for the implementation vulns found by RHUL - and we should hopefully conclude that next week by moving everything into the matrix-rust-sdk crypto create implmentation: https://github.com/vector-im/element-web/issues/21972#issuec... is the tracker.

Then, we can address the harder server-controlled group membership issue in one place. First step will be to improve device verification & trust so that trust is the default, not the exception, to make it easier to spot and warn about unexpected devices in the room. The full solution is then either MSC3917 (https://github.com/matrix-org/matrix-spec-proposals/blob/fay...) - or potentially to switch everything to MLS.

We're working on MLS anyway in parallel to RHUL mitigation work; you can see the progress at https://arewemlsyet.com, and it's looking good.

I'm guessing you're not interested in doing a podcast on "yay we converged our crypto implementations on a single robust Rust implementation so we can fix the remaining bugs in one place", but as soon as the server-controlled group membership thing is solved we'll be in touch. Work has also gone much slower than hoped on this, thanks to the joys of funding open source.

https://github.com/matrix-org/matrix-spec-proposals/blob/keg... is how we’re doing it, and it’s being implemented currently in Dendrite.

Matrix already has key-based identity in the works at https://github.com/matrix-org/matrix-spec-proposals/blob/keg... (and implemented in Dendrite at https://github.com/matrix-org/dendrite/pulls?q=is%3Apr+is%3A...). Matrix is set up to let folks go wild and change fundamentals like this; basically every Matrix Spec Change (MSC) is a small fork, which then gets merged into the main spec if it can be proven to work well in the wild.

Gitter seems to have moved to being a Matrix instance (or maybe it always has? it didn't look like Matrix when I used it circa 2016), but matrix feels half-baked and is just a bunch of hacks put together. For example

- Can't "mark all as read" on a space. probably because rooms within a space are only tangentially related,

- No custom emojis or sticker packs (their proposal for this is to create rooms to house custom emojis/sticker packs[0])

Not a great bet to go to keybase with the Zoom acquisition https://news.ycombinator.com/item?id=28814210

0: https://github.com/matrix-org/matrix-spec-proposals/pull/195...

We’re currently working on account portability (https://github.com/matrix-org/matrix-spec-proposals/pull/401...) and experimenting with glueing bluesky style DIDs onto it (so as to provide DMs for bluesky via Matrix, should they want them)

slow moving but there is discussions https://github.com/matrix-org/matrix-spec-proposals/pull/1998