seshat VS matrix-js-sdk

> Can you already say something about encrypted search support in Element X?

It's planned, and should be way easier to do than on old Element given we can hook the clientside search indexer (https://github.com/matrix-org/seshat), which happens to be in Rust, directly into matrix-rust-sdk and thence both EIX and EAX. Whereas previously we'd have had to do custom integs with Seshat on EI and EA like the current one on Element Desktop, which just becomes a maintenance nightmare. Hopefully should be ready in the next months.

> Also since you mentioned it, is multiaccount support something that we can realistically expect to see in Element X someday (maybe even in a v1.0?). If I remember correctly for the previous iterations of the clients the status was mostly "maybe one day" or "the architecture does not really support it".

Definitely. We actually had it pre-Element in Matrix Console, and then it got lost in Element. The hope is to get it into Element X relatively early.

https://github.com/matrix-org/seshat - we’ve had it for years on desktop; just need to hook it up on mobile. it’s currently blocked behind unifying sdks, so we don’t have to do it twice in iOS & Android.

Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.

The competitive gap with Discord in terms of media quality is probably something like:

* Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, and letting anyone bring their SFU to the party.

* Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/

* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too.

* Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important.

We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.

We can probably also be more bullish about supporting new audio codecs like Lyra.

Messages are stored in encrypted form on the Beeper server and the Beeper client has a local search index (the same one used by Element desktop: https://github.com/matrix-org/seshat)

I'd quite like some more high level documentation for the Matrix JS SDK (https://github.com/matrix-org/matrix-js-sdk). I've been looking at it for quite some time and still don't understand how timelines work.

Currently we don't force TURN, so in practice this means that voice packets go direct between the clients if possible, and so the IP addresses of the clients are necessarily exposed to each other.

However, this is utterly trivial to fix: matrix-js-sdk already exposes https://github.com/matrix-org/matrix-js-sdk/blob/96ba061732b... and we simply haven't exposed it as a setting in Element Call yet. I've filed a bug for it at https://github.com/vector-im/element-call/issues/251 - thanks for bringing it up!

In terms of moderation: this is no different to moderation in Matrix as a whole, where we're already busy working on shared greylists (MSC2313 and friends) - https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix... has more details at the end.

We expect to turn it on in the next few weeks - the problem we were debugging that we meant it was turned off looks to be fixed. https://github.com/matrix-org/matrix-js-sdk/pull/2002 is the PR that will need to merge to enable it ftr.

Unlike any open source projects to which I've contributed, this project involves 2 other repos, matrix-react-sdk and matrix-js-sdk. As explained in the Development guide in the element-web repo, I need those 2 SDKs in order to build Element successfully for code contribution.