Tokenizing / picking words out of non-english languages

This page summarizes the projects mentioned and recommended in the original post on /r/LanguageTechnology
nlp-library morphological-analysis Segmentation pos-tagging

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • SudachiPy

    Discontinued Python version of Sudachi, a Japanese tokenizer.

    • spaCy uses SudachiPy internally (see the doc comment about that), so if you don't need any of spaCy's extra features or want more control over the tokenization, you could use SudachiPy directly.

  • alphabet-soup

    Alphabet Soup gives language learners easily digestible chunks for practice.

    • By the way, where did you source your "Japanese database containing linguistic information"? I use Tatoeba and Aozora Bunko for example sentences and JMdict/EDICT for my project Alphabet Soup. (Also Kuromoji for tokenization.)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Python Text Parsing Project: Furigana Inserter for Anki

    2 projects | dev.to | 31 Aug 2021

  • software which turn hiragana and katakana into kanji

    1 project | /r/LearnJapanese | 29 Aug 2021

  • Gauging interest and plausibility of an overhaul of Anki's Morphman

    2 projects | /r/LearnJapanese | 31 Dec 2020

  • [Arabic>latin transliteration] any apps for this?

    1 project | /r/translator | 30 Apr 2023

  • Sakubun - a tool I made to help you practice kanji, with customized quiz questions and sentences

    3 projects | /r/LearnJapanese | 1 Sep 2022