lingua VS cld3

select 'r/'||subreddit sub , initcap(lang) language , count(*) c , ratio_to_report(c) over(partition by sub) ratio , sum(iff(language!='English', c, 0)) over(partition by sub) total_not_english , sum(c) over(partition by sub) total from reddit_sample_languages_udtf group by 1, 2 qualify ratio > .02 order by total_not_english desc, c desc, 1, ratio desc- Jason Baumgartner for collecting and sharing Reddit’s comments. - Peter M. Stahl for the Lingua project to detect languages in Java. - Snowflake for making it easy to run Java code in a UDF.

I don't speak most of these languages, so I wasn't able to verify -- instead I just used the results of this library: https://github.com/pemistahl/lingua

A linguist can infer the language of the field. It's also possible to use an automated process in the pipeline. A couple of NLP libraries are available in the JVM ecosystem, but I set my eyes on Lingua, one focused on language recognition.

I've used this one previously, and found it pretty easy to use, relatively fast, and accurate: https://github.com/pemistahl/lingua

the C++ code is from https://github.com/google/cld3

Python is widely used in natural language processing, so there are a couple of comprehensive open source libraries for this task, such as Google's CLD 2 and CLD 3, langid and langdetect. Unfortunately, except for the last one they have two major drawbacks:

I was looking for something like that for .net app and ended up using this https://github.com/google/cld3

I used it a lot in the models I work on. For example, we used it on a widely used project called CLD3 (https://github.com/google/cld3). The performance is great.