NLP-progress VS SymSpell

A standard step in Chinese text processing is word segmentation, which deals with this problem.

It would also be cool to have some charts with parameter density and even overall effectiveness (a tl;dr version of SOTA-trackers, maybe?) if that doesn't prove too infeasible.

Another great resource for keeping tabs on the state of the art performance for common tasks is: NLP Progress

For someone who needs to be on top of the latest research - Twitter (distraction-prone, marketing-friendly, instantly-gratifying, quick), newsletters in ML + NLP (https://jack-clark.net/, ruder.io, offconvex.org, etc.) (distraction-free, generic, time-consuming), SOTA chasing (https://paperswithcode.com/, http://nlpprogress.com/) (distraction-free, generic + focused, code-friendly)

NLP is a very broad topic and like you said it can be extremely overwhelming to keep up with all the recent advancements, especially if you are a beginner. I would suggest you to take a look at nlp_tasks or NLP-progress or The Big Bad NLP Database to get an idea of the different tasks in NLP and see if you can find anything that looks interesting to you.

One place to start is nlp progress if leader boards are your thing, if the model on top of the leader board is not a transformer based model and one further down is, you have your answer.

Books: a. "Speech and Language Processing" by Daniel Jurafsky and James H. Martin (3rd Edition) - This book covers various aspects of natural language processing, including a section on spelling correction that provides a comprehensive introduction to the topic. b. "Foundations of Statistical Natural Language Processing" by Christopher D. Manning and Hinrich Schütze - This book provides an overview of statistical approaches in NLP, including a chapter on spelling correction. Articles: a. "How to Write a Spelling Corrector" by Peter Norvig - This article demonstrates the development of a simple spelling corrector using statistical algorithms. It's a great starting point for understanding the basics of spell checkers. (Link: https://norvig.com/spell-correct.html) b. "The Design of a Proofreading Software Service" by Michael D. Garris and James L. Blue - This article presents the design and implementation of a spelling correction system that can be integrated into various applications. (Link: https://www.nist.gov/system/files/documents/itl/iad/89403123.pdf) c. "A Fast and Flexible Spellchecker" by Atkinson, K. (2006) - This article details the design of a spell checker that uses a combination of rule-based and statistical approaches for improved performance. (Link: https://aspell.net/0.60.6.1/aspell-0.60.6.1.pdf) Online Resources: a. The Natural Language Toolkit (NLTK) - This is a popular Python library for natural language processing. It includes a spell checker module and various examples of how to use it. (Link: https://www.nltk.org/) b. SymSpell - This is an open-source spell checking library that uses a Symmetric Delete spelling correction algorithm for high performance and accuracy. The GitHub repository includes a detailed description of the algorithm and examples of how to use it. (Link: https://github.com/wolfgarbe/SymSpell) These resources should provide a solid foundation for understanding the design, algorithms, and usage of spell checkers. Happy learning!

Can this github.com/wolfgarbe/SymSpell or this github.com/ruby/did_you_mean or any of these github.com/topics/spell-check?o=desc&s=forks spellcheckers be used as an autocorrection software?

SymSpell: 1M times faster spelling correction\ (6 comments)

SymSpell

I'm glad to see these built-in to Postgres, as these are the basics of fuzzy string matching.

A quantum leap would be to integrate an implementation of the symmetric delete algorithm, such as https://github.com/wolfgarbe/SymSpell

Soundex and Phonex can yield too many false negatives outside of phonetically English names. Levenshtein/Jaro-Winkler aren't indexable solutions themselves, so they require N^2 comparisons. SymSpell conceptually combines these two into an indexed string-distance solution. It has the usual index issue of being designed for many reads, few writes.

Some of the modern approaches to spellchecking still take this road: for example, SymSpell algorithm (claiming to be "1 million times faster") is at its core just a brilliant idea for a novel storage format for a flat word list, that allows optimizing the calculation of edit distance significantly.

https://github.com/wolfgarbe/SymSpell lists 5 JS implementations (+ a Rust one that compiles to web assembly)