Text processing

Open-source projects categorized as Text processing | Edit details
Language filter: + Python + Go + Rust + Shell + Java

Top 23 Text processing Open-Source Projects

  • ripgrep

    ripgrep recursively searches directories for a regex pattern while respecting your gitignore

    Project mention: What programming language would you suggest learning to someone who likes PowerShell? | reddit.com/r/PowerShell | 2022-01-18

    ripgrep over grep I love regex

  • micro-editor

    A modern and intuitive terminal-based text editor

    Project mention: Simple text file creation. | reddit.com/r/linux | 2022-01-17

    which creates an empty file by the name filename.txt. then you edit its contents however you want (for a starter-friendly command-line text editor, I recommend micro.)

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • GoQuery

    A little like that j-thing, only in Go.

    Project mention: Building Golang crawler with Docker | reddit.com/r/golang | 2021-03-12

    RUN go get github.com/PuerkitoBio/goquery

  • Command-line-text-processing

    :zap: From finding text to search and replace, from sorting to beautifying text and more :art:

    Project mention: My simple GitHub project went Viral | news.ycombinator.com | 2021-04-14

    I had a similar experience with one of my GitHub repos [0] that is currently 9k+ stars. I added donation link when it was about 5k stars (after it went viral courtesy HN). But this was before GitHub sponsors. I removed donation links after I got only a single donation in about a year.

    I had much better results when I started converting my tutorials into ebooks and sold them. Obviously having a paid product is different, but I'm referring to the paid sales I got whenever I put up 'pay what you want' offer.

    [0] https://github.com/learnbyexample/Command-line-text-processi...

  • pydantic

    Data parsing and validation using Python type hints

    Project mention: Strict Python Function Parameters | news.ycombinator.com | 2022-01-23

    Slightly off-topic, but everyone writing modern Python should be familiar with Pydantic and similar libraries that use type hints for validation and parsing:




    We're using Pydantic for Robusta (https://github.com/robusta-dev/robusta) and absolutely love it. You get the best of traditional Python (rapid prototyping and no boilerplate) while still being able to scale your codebase and keep it maintainable. Robusta is the first large project I've written in Python where I'm not encountering type errors at runtime left and right.

  • fuzzywuzzy

    Fuzzy String Matching in Python

    Project mention: I made a bot that stops muck chains, here are the phrases that he looks for to flag the comment as a muck comment. Are there any muck forms I forgot about? | reddit.com/r/DaniDev | 2021-12-08

    You can have a look at this library to use fuzzy search instead of looking for plaintext muck: https://github.com/seatgeek/fuzzywuzzy

  • diff-match-patch

    Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

    Project mention: Keeping track of changes made to xml file. | reddit.com/r/learnprogramming | 2021-10-18

    A bit late to the party but have you checked this? google/diff-match-patch

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • blackfriday

    Blackfriday: a markdown processor for Go

    Project mention: Compounding Competence | dev.to | 2021-04-11

    On the backend when generating the emails: For this, I chose a popular Go markdown library BlackFriday.

  • sh

    A shell parser, formatter, and interpreter with bash support; includes shfmt (by mvdan)

    Project mention: Indenting piped shell expressions in a script? | reddit.com/r/bash | 2022-01-11

    I also like running shfmt over my shell scripts so they all look the same without me having to think about whitespace.

  • toml

    TOML parser for Golang with reflection. (by BurntSushi)

    Project mention: Rust Moderation Team Resigns | news.ycombinator.com | 2021-11-22

    He's also a prominent contributor to the Go ecosystem.


  • 汉字拼音转换工具(Python 版)


  • ftfy

    Fixes mojibake and other glitches in Unicode text, after the fact.

  • go-humanize

    Go Humans! (formatters for units to human friendly sizes)

  • Lark

    Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

    Project mention: Made a Programing language using python | reddit.com/r/Python | 2021-11-29

    There's also lark, which is used by a plethora of projects (I haven't used it, but I heard about PreQL on a podcast where they talk for a bit about what it's like to develop a new language in lark)

  • phonenumbers

    Python port of Google's libphonenumber

    Project mention: Does anyone know where I can find official docs for python-phonenumbers package? | reddit.com/r/learnprogramming | 2022-01-12

    This is the GitHub repo for the package.

  • sd

    Intuitive find & replace CLI (sed alternative)

    Project mention: Useful sed scripts & patterns. | reddit.com/r/commandline | 2021-11-12

    Have you ever compared sed with sd? https://github.com/chmln/sd

  • sqlparse

    A non-validating SQL parser module for Python

    Project mention: Open Source SQL Parsers | dev.to | 2021-10-08

    Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.

  • TextDistance

    Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.

    Project mention: life4/textdistance: Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage. | reddit.com/r/Python | 2021-09-06
  • Java String Similarity

    Implementation of various string similarity and distance algorithms: Levenshtein, Jaro-winkler, n-Gram, Q-Gram, Jaccard index, Longest Common Subsequence edit distance, cosine similarity ...

  • regex

    An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.

    Project mention: Added std::regex to a regex shootout and the results were surprising | reddit.com/r/cpp | 2022-01-04

    That one rust_regex result being so much slower than everything else irked me, so I filed an for it here: https://github.com/rust-lang/regex/issues/827

  • bluemonday

    bluemonday: a fast golang HTML sanitizer (inspired by the OWASP Java HTML Sanitizer) to scrub user generated content of XSS

    Project mention: HTML Sanitizer API | news.ycombinator.com | 2021-05-06

    My thoughts as a maintainer of a HTML sanitizer https://github.com/microcosm-cc/bluemonday

    1. Sanitizing is not difficult, defining the policy/config is difficult as your need is not someone else's. First glance of this proposal is that this needs a lot more work to cover people's needs. It's good enough, but will have a lot of edges and will need to evolve.

    2. If you allow a blocklist then people will use that by default as it's easier to say "I don't want " than it is to say "I only accept 3. Even if you sanitize something you should keep the raw input... you should store the raw input alongside the sanitized (in fact the sanitized is merely a cached version of the raw input having been sanitized). The reason for this is you will have issues you need to debug (and can't without the input) and you will have round-trip edits you should support (but it's not round-trippable when everything you return is different from the input, do not punish a user who pasted HTML thinking it was safe by then not allowing them to edit it out because you threw everything away). Additionally if you want to ever report on the input, i.e. topK values, and you've modified the input and not kept raw, then you can never do this.

    4. Provide a sane default. Most engineers simply do not know what is safe or not. I ship a policy in bluemonday for user generated content... it is safe by default and good enough for most people, and it can be taken and extended due to the way the API is structured so can cover other scenarios as a foundation policy.

    I think the proposal in general: specify a standard for a sanitization API has merit. But mostly it has merit if it specifies a standard for defining sanitization policies/configuration, allowing them to be portable across different languages and systems.

    The one I wrote is very heavily inspired by https://github.com/owasp/java-html-sanitizer which is the OWASP project one maintained by Mike Samuel. When I did my research before writing the Go one, this was far and away the best way to construct the policy/config and I already saw that this perspective was more valuable than whether it's a token based parser (GIGO but low memory) or a DOM builder (more memory)... no-one cares about the internals, they care about expressing what safe means to them.

  • PLY

    Python Lex-Yacc

  • gofeed

    Parse RSS, Atom and JSON feeds in Go

    Project mention: Automatice el README para su perfil de GitHub con Go y GitHub Actions | dev.to | 2021-04-25
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-01-23.

Text processing related posts


What are some of the best open-source Text processing projects? This list will help you:

Project Stars
1 ripgrep 28,992
2 micro-editor 18,705
3 GoQuery 11,046
4 Command-line-text-processing 9,793
5 pydantic 8,781
6 fuzzywuzzy 8,587
7 diff-match-patch 5,003
8 blackfriday 4,862
9 sh 4,415
10 toml 3,727
11 汉字拼音转换工具(Python 版) 3,665
12 ftfy 3,178
13 go-humanize 3,007
14 Lark 2,969
15 phonenumbers 2,916
16 sd 2,859
17 sqlparse 2,696
18 TextDistance 2,590
19 Java String Similarity 2,415
20 regex 2,159
21 bluemonday 2,158
22 PLY 2,090
23 gofeed 1,788
Find remote jobs at our new job board 99remotejobs.com. There are 29 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
OPS - Build and Run Open Source Unikernels
Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.