laserembeddings VS marian-dev

You're pretty much right on the money. For ParaCrawl[1] (which I worked on) we used fast machine translation systems that were "good enough" to translate one side of each pair to the language of the other, see whether they'd match sufficiently, and then deal with all the false positives through various filtering methods. Other datasets I know of use multilingual sentence embeddings, like LASER[2], to compute the distance between two sentences.

Both of these methods have a bootstrapping problem, but at this point in the MT for many languages we have enough data to get started. Previous iterations of ParaCrawl used things like document structure and overlap of named entities among sentences to identify matching pairs. But this is much less robust. I don't know how they solve this problem today for low-resource languages.

[1] https://paracrawl.eu

[2] https://github.com/yannvgn/laserembeddings

I've been using LASER from Facebook Research via https://github.com/yannvgn/laserembeddings to accept multi-lingual input in front of the the domain-specific models for recommendations and stuff (that are trained on English annotated examples).

As far as I could tell this is the engine powering this:

https://github.com/marian-nmt/marian-dev/

It’s compiled c++ running using wasm.

This is the underlying translation engine:

https://github.com/marian-nmt/marian-dev

And the browser fork of it:

https://github.com/browsermt/marian-dev

You can find the engine used here [1], the API built around it here [2] and its WASM port here [3] and the WebAssembly matrix multiplication optimizations are here [4]

[1] https://marian-nmt.github.io/