Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Python Text processing Projects
-
First, note the method prefix_allowed_tokens_fn. This method applies a Pydantic model to constrain/guide how the LLM generates tokens. Next, see how that constrain can be applied to txtai's LLM pipeline.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
diff-match-patch
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.
Project mention: Ideas for approaching pattern matching/distance problem | /r/learnprogramming | 2023-06-29I also came across this diff match algorithms: https://github.com/google/diff-match-patch
-
Could probably whip up a python script real quick with this library: https://github.com/mozillazg/python-pinyin. Probably need some extra logic to deal with heteronyms. Not sure what your goal is.
-
Lark
Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
Project mention: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python | news.ycombinator.com | 2023-08-13Lark supports, and recommends, writing and storing the grammar in a .lark file. We have syntax highlighting support in all major IDEs, and even in github itself. For example, here is Lark's built-in grammar for Python: https://github.com/lark-parser/lark/blob/master/lark/grammar...
You can also test grammars "live" in our online IDE: https://www.lark-parser.org/ide/
The rationale is that it's more terse and has less visual clutter than a DSL over Python, which makes it easier to read and write.
-
-
Project mention: Show HN: Databasediagram.com – Private, Text to Entity-Relationship Diagram Tool | news.ycombinator.com | 2023-06-08
Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
TextDistance
📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
-
-
After over a year since the last release of pyparsing, I've bundled up all the bug-fixes and changes, and they are now released as pyparsing 3.1.0. Visit this link for the details.
-
-
Project mention: Ask HN: Could you show your personal blog here? | news.ycombinator.com | 2023-07-04
Unlike many people here, I don't like to write hundreds of mediocre posts. Instead, I prefer very few posts, that unfortunately are still mediocre.
If you're tired of all the perfection that exists on the internet, where every piece is deeply insightful and changes your life, I'd encourage you to read my articles, which only promise to shorten it:
-
msgspec
A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML
Project mention: Htmx, Rust and Shuttle: A New Rapid Prototyping Stack | news.ycombinator.com | 2023-11-01 -
-
-
python-user-agents
A Python library that provides an easy way to identify devices like mobile phones, tablets and their capabilities by parsing (browser) user agent strings.
-
Project mention: LongRoPE: Extending LLM Context Window Beyond 2M Tokens | news.ycombinator.com | 2024-02-22
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
-
btw there's also python's native pyfiglet https://github.com/pwaller/pyfiglet
-
Construct
Construct: Declarative data structures for python that allow symmetric parsing and building
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2024-04-18.
Python Text processing related posts
- Advanced RAG with guided generation
- LongRoPE: Extending LLM Context Window Beyond 2M Tokens
-
utype VS pydantic - a user suggested alternative
2 projects | 15 Feb 2024
- Pydantic v2 ruined the elegance of Pydantic v1
- Ask HN: Pydantic has too much deprecation. Why is it popular?
- OpenAI uses Pydantic for their ChatCompletions API
- If you're late, consider creating your CV with this Python code: RenderCV
-
A note from our sponsor - InfluxDB
www.influxdata.com | 18 Apr 2024
Index
What are some of the best open-source Text processing projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | pydantic | 18,521 |
| 2 | fuzzywuzzy | 9,067 |
| 3 | diff-match-patch | 7,080 |
| 4 | 汉字拼音转换工具（Python 版） | 4,666 |
| 5 | Lark | 4,471 |
| 6 | ftfy | 3,704 |
| 7 | sqlparse | 3,574 |
| 8 | phonenumbers | 3,398 |
| 9 | TextDistance | 3,296 |
| 10 | PLY | 2,695 |
| 11 | pyparsing | 2,083 |
| 12 | chardet | 2,071 |
| 13 | shortuuid | 1,975 |
| 14 | msgspec | 1,839 |
| 15 | python-slugify | 1,446 |
| 16 | typeguard | 1,432 |
| 17 | python-user-agents | 1,404 |
| 18 | DataProfiler | 1,357 |
| 19 | pyfiglet | 1,299 |
| 20 | Construct | 875 |
| 21 | xpinyin | 809 |
| 22 | python-nameparser | 635 |
| 23 | Charset Normalizer | 517 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com