Top 23 Python Natural Language Processing Projects

• transformers

  12 39,664 9.9 Python

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.

  Latest mention: Replicating GPT-2 at Home | news.ycombinator.com | 2021-01-23

  4. Repeat

  For step 3 you need to send the gradients from each GPU somewhere, and then send back either the averaged gradient or the updated model weights. So when the model is large (say, 3GB for GPT 774M!) that's a lot of GPU-GPU communication!

  You're right that for the vast majority of ML cases, the models are small enough that the synchronization cost is negligible, though.

  I wrote up some benchmarks here:

  https://github.com/huggingface/transformers/issues/9371

• funNLP

  0 28,070 3.7 Python

  中英文敏感词、语言检测、中外手机/电话归属地/运营商查询、名字推断性别、手机号抽取、身份证抽取、邮箱抽取、中日文人名库、中文缩写库、拆字词典、词汇情感值、停用词、反动词表、暴恐词表、繁简体转换、英文模拟中文发音、汪峰歌词生成器、职业名称词库、同义词库、反义词库、否定词库、汽车品牌词库、汽车零件词库、连续英文切割、各种中文词向量、公司名字大全、古诗词库、IT词库、财经词库、成语词库、地名词库、历史名人词库、诗词词库、医学词库、饮食词库、法律词库、汽车词库、动物词库、中文聊天语料、中文谣言数据、百度中文问答数据集、句子相似度匹配算法集合、bert资源、文本生成&摘要相关工具、cocoNLP信息抽取工具、国内电话号码正则匹配、清华大学XLORE:中英文跨语言百科知识图谱、清华大学人工智能技术系列报告、自然语言生成、NLU太难了系列、自动对联数据及机器人、用户名黑名单列表、罪名法务名词及分类模型、微信公众号语料、cs224n深度学习自然语言处理课程、中文手写汉字识别、中文自然语言处理 语料/数据集、变量命名神器、分词语料库+代码、任务型对话英文数据集、ASR 语音数据集 + 基于深度学习的中文

• jieba

  0 25,296 0.1 Python

  结巴中文分词

• spaCy

  0 18,095 9.2 Python

  💫 Industrial-strength Natural Language Processing (NLP) with Python and Cython

• gensim

  1 11,612 8.5 Python

  Topic Modelling for Humans

  Latest mention: Koan: A word2vec negative sampling implementation with correct CBOW update | news.ycombinator.com | 2021-01-02

  Apparently it did: https://github.com/RaRe-Technologies/gensim/issues/1873

• nltk

  1 9,563 7.8 Python

  NLTK Source

  Latest mention: Wordnet and Sexism | reddit.com/r/datascience | 2021-01-03

• bert-as-service

  2 8,791 3.2 Python

  Mapping a variable-length sentence to a fixed-length vector using BERT model

  Latest mention: Needed 100% to pass a safety quiz, need to wait a week to retake | reddit.com/r/mildlyinfuriating | 2021-01-12

  You joke but

• pattern

  0 7,755 0.4 Python

  Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

• textblob

  0 7,489 3.9 Python

  Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase extraction, translation, and more.

• datasets

  3 6,551 9.8 Python

  🤗 The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools

  Latest mention: [P] 611 text datasets in 467 languages in the new v1.2 release of HuggingFace datasets library | reddit.com/r/MachineLearning | 2021-01-05

  There will be 13 more bytthe end of this week, from Microsoft CodeXGlue, I had not the time to fix my PR earlier : https://github.com/huggingface/datasets/pull/997 .

• ciphey

  6 6,160 9.7 Python

  ⚡ Automatically decrypt encryptions without knowing the key or cipher, decode encodings, and crack hashes ⚡

  Latest mention: Need Assistance In Decrypting | reddit.com/r/encryption | 2021-01-15

  If you want a code cracked, in case of #2, there are many resources online and offline that can help you solve this. One of them is the excellent /r/codes subreddit, which may be able to point you in the right direction. Also interesting is Ciphey, which can identify and decrypt a lot of codes.

• pytext

  0 6,118 9.2 Python

  A natural language modeling framework based on PyTorch

• snownlp

  0 5,241 0.0 Python

  Python library for processing Chinese text

• pkuseg-python

  0 5,240 2.7 Python

  pkuseg多领域中文分词工具; The pkuseg toolkit for multi-domain Chinese word segmentation

• stanza

  0 5,119 8.8 Python

  Official Stanford NLP Python Library for Many Human Languages

• doccano

  0 4,188 9.0 Python

  Open source text annotation tool for machine learning practitioner.

• PyTorch-NLP

  0 1,851 5.5 Python

  Basic Utilities for PyTorch Natural Language Processing (NLP)

• polyglot

  0 1,750 1.2 Python

  Multilingual text (NLP) processing toolkit

• langid.py

  0 1,707 0.0 Python

  Stand-alone language identification system

• uda

  1 1,594 0.4 Python

  Unsupervised Data Augmentation (UDA)

  Latest mention: A Visual Survey of Data Augmentation in NLP | dev.to | 2020-08-26

  The words that replaces the original word are chosen by calculating TF-IDF scores of words over the whole document and taking the lowest ones. You can refer to the code implementation for this in the original paper here.

• textacy

  0 1,592 8.2 Python

  NLP, before and after spaCy

• quepy

  0 1,197 0.0 Python

  A python framework to transform natural language questions to queries in a database query language.

• TextAttack

  1 1,192 9.7 Python

  TextAttack 🐙 is a Python framework for adversarial attacks, data augmentation, and model training in NLP

  Latest mention: A Visual Survey of Data Augmentation in NLP | dev.to | 2020-08-26

  Libraries like nlpaug and textattack provide simple and consistent API to apply the above NLP data augmentation methods in Python. They are framework agnostic and can be easily integrated into your pipeline.

Index