apertium VS lingva-translate

This is very cool, looking forward to it! I've been doing the same thing with Spanish Wikipedia articles for a while, using a few lines of Bash + Regex. I was using Apertium for it. https://apertium.org/ It's definitely worse than most ML-based solutions, but it works reliably and fast; you can run it entirely offline. With Spanish translations, the main problem I was facing is lack of vocabulary, so I created https://github.com/phil294/apertium-eng-spa-wiktionary which about doubles the amount of recognized words, albeit with wonky grammar.

I used to keep track of the state of machine translation some years back.

I think the way you measure the success of an automated translation is edit distance, i.e. how many manual edits you need to make to a translated text before you reach some acceptable state. I suppose it's somewhat subjective, but it is possible to construct a benchmark and allow for multiple correct results.

The best resources I knew back then were:

VISL's CG-3 self-reported a competitively low edit distance compared to Google Translate: https://visl.sdu.dk/constraint_grammar.html -- the abstraction unfortunately requires a rather deep knowledge of any one particular language's grammar. It is a convincing argument that in order to beat Google Translate, you want less fuzzy machine learning and more structural analysis. But you also need a PhD in computational linguistics and deep knowledge of each language.

Apertium has an open-source pipeline: https://apertium.org/ -- seems to be much more like an open-source approach with a quality similar to Google Translate (although I don't know if it's better or worse; probably slightly worse in most cases, and with a slightly lower coverage).

For context, I have been contributing CV Creole data to Unicode's CLDR and MediaWiki for a number of years now, but both are mostly manual work. I once considered setting up an Apertium language pair between CV Creole and Portuguese, given the grammatical similarities, but never got around to it.

Lingva is awesome. Also don't forget to check out LibreTranslate and Apertium. They are open source. Apertium can even translate web pages (you need to enter the URL).

#!/bin/bash set -x mkdir -p apertium-src && \ mkdir -p $MTDIR cd apertium-src && \ wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-8.5.0/gcc-8.5.0.tar.gz -O - \ | gzip -dc \ | tar -xf - && \ cd gcc-8.5.0 && \ ./configure --prefix=$MTDIR --disable-multilib && \ make -j $(nproc) && \ make install && \ cd .. || exit 1 cd apertium-src && \ wget https://github.com/unicode-org/icu/releases/download/release-69-1/icu4c-69_1-src.tgz -O - \ | gzip -dc \ | tar -xf - \ && cd icu/source \ && CC=gcc CXX=g++ ./configure --prefix=$MTDIR \ && CC=gcc CXX=g++ make -j $(nproc) \ && CC=gcc CXX=g++ make install \ && cd ../.. \ || exit 1 cd apertium-src && \ svn checkout http://beta.visl.sdu.dk/svn/visl/tools/vislcg3/trunk vislcg3 && \ cd vislcg3 && ./get-boost.sh \ && ./cmake.sh -DCMAKE_INSTALL_PREFIX=$MTDIR \ -DICU_INCLUDE_DIR=$MTDIR/include \ -DICU_LIBRARY=$MTDIR/lib/libicuuc.so \ -DICU_IO_LIBRARY=$MTDIR/lib/libicuio.so \ -DICU_I18N_LIBRARY=$MTDIR/lib/libicui18n.so \ && make -j$(nproc) && \ make install && cd .. || exit 1 cd apertium-src && \ git clone https://github.com/apertium/lttoolbox && \ cd lttoolbox && ./autogen.sh --prefix=$MTDIR && make -j $(nproc) && make install && cd ../.. || exit 1 cd apertium-src && \ git clone https://github.com/apertium/apertium && \ cd apertium && ./autogen.sh --prefix=$MTDIR && make -j $(nproc) && make install && cd ../.. || exit 1 cd apertium-src && \ git clone https://github.com/apertium/apertium-lex-tools && \ cd apertium-lex-tools && ./autogen.sh --prefix=$MTDIR && make -j $(nproc) && make install && cd ../.. || exit 1 cd apertium-src && \ git clone https://github.com/apertium/apertium-tha && \ cd apertium-tha && ./autogen.sh --prefix=$MTDIR && make && make install && cd ../.. || exit 1 cd apertium-src && \ git clone https://github.com/apertium/apertium-tha-eng && \ cd apertium-tha-eng && ./autogen.sh --prefix=$MTDIR && make && make install && cd .. && \ cd .. || exit 1

Interestingly, I think this is actually related to the offline translation features built into Firefox. Both are products of "Project Bergamot", but the Mozilla-maintained version was later merged into the Firefox application:

https://browser.mt/

https://blog.mozilla.org/en/mozilla/local-translation-add-on...

https://hacks.mozilla.org/2022/06/training-efficient-neural-...

https://github.com/mozilla/firefox-translations

https://firefox-source-docs.mozilla.org/toolkit/components/t...

Extra webpage with screenshot and links, impossible to search for normally:

https://translatelocally.com/downloads/

Does one thing and does it well.

Oh— For downloading models, it's much easier to pipe/`xargs` `translateLocally --available-models` into `translateLocally -d` than go through the GUI.

---

Other self-hostable translation tools:

https://www.apertium.org/index.eng.html

- Traditional rule-based translation. Seems to work pretty well, but no good desktop frontend.

https://www.argosopentech.com/

- Works, but crashy desktop app.

https://libretranslate.com/

- API wrapping Argos Translate.

https://lingva.thedaviddelta.com/

- Google Translate scraper/privacy frontend.

https://euroglot.com/

- Proprietary, subscription trialware.

Google Translate => Lingva

I use https://github.com/thedaviddelta/lingva-translate . I now is not offline solution and is a front-end for Google Translate but retrieves the translation without directly accessing any Google-related service, preventing them from tracking . Is similiar to NewPipe

Lingva

Can you choose online service like Lingva translate?

You could use https://lingva.ml/ (a proxy frontend for google translate) or a Firefox container for google translate only