[Computer Stuff] What's the best way to split a Japanese sentence into "words"?

• janome

  2 828 5.2 Python

  Japanese morphological analysis engine written in pure Python

I did program stuff like that a bit in Korean and Japanese. So, in short, these tools/libraries are called 'Tokenizers'. I.e. search for "Japanese tokenizer", it will also tell you that MeCab is one of them. There is no good/easy way to split words in Japanese with simple algorithms, so these libraries, that are based on statistics or AI, will be your only choice. There is a good example sentence that shows how futile this would be without those libraries: "すもももももももものうち". From here.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project