doctr VS PaddleOCR

checkout https://github.com/mindee/doctr or https://github.com/VikParuchuri/surya for something practical

multimodal llm would of course blow it all out the water, so some llama3-like model is probably SOTA in terms of what you can run yourself. something like https://huggingface.co/blog/idefics2

https://github.com/mindee/doctr/issues/1049

I am looking for something this polished and reliable for handwriting, does anyone have any pointers? I want to integrate it in a workflow with my eink tablet I take notes on. A few years ago, I tried various models, but they performed poorly (around 80% accuracy) on my handwriting, which I can read almost 90% of the time.

Yup! But I'm still exploring options. (any recommendations would be welcomed!) Here are some candidates I'm considering:

- https://github.com/mindee/doctr

- https://github.com/open-mmlab/mmocr

- https://github.com/PaddlePaddle/PaddleOCR (honestly I don't know Mandarin so I'm a bit stuck)

- https://github.com/clovaai/donut - While it's primarily an "OCR-free document understanding transformer," I think it's worth experimenting with. Think I can sort this out by letting the LLM reason through it multiple times (although this will impact performance)

- yesterday got a suggestion to consider https://github.com/kakaobrain/pororo - I don't think development is still active but the results are pretty great on Korean text

EasyOCR is a popular project if you are in an environment where you can use run Python and PyTorch (https://github.com/JaidedAI/EasyOCR). Other open source projects of note are PaddleOCR (https://github.com/PaddlePaddle/PaddleOCR) and docTR (https://github.com/mindee/doctr).

Last I checked I saw a grocery bill example using https://github.com/mindee/doctr and was fairly accurate. Bear in mind that was last year, hopefully it got even better or there are other libraries

For its OCR service, Cape uses the excellent Python docTR library. Some of the critical benefits of docTR are its ease of use, flexibility, and matching state-of-the-art performance. The OCR model consists of two steps: text detection and text recognition. Cape uses a pre-trained DB Resnet50 architecture for detection, and for recognition, it uses a MobileNetV3 Small architecture. To learn more about the level of OCR accuracy you can expect for your document, you can consult these benchmarks provided by docTR. As you will see, model performance is very competitive compared to other commercial services.

There's also DocTR which can do text detection and extraction out of the box.

It's command line driven but can display the detected text as an overlay of the document.

https://github.com/mindee/doctr

If you want to OCR a document image, modern versions of Tesseract can work well. If you last used it a few years ago, the recognition has improved since due to a new text recognition algorithm that uses modern (deep learning) techniques. Browser demo using a modern version: https://robertknight.github.io/tesseract-wasm/.

OCR processing typically consist of two major steps: detecting/locating words or lines of text on the page, and recognizing lines of text.

Tesseract's text recognition uses modern methods, but the text detection phase is still based on classical methods involving a lot of heuristics, and you may need to experiment with various configuration variables to get the best results. As a result it can fail to detect text if you present it with something other than a reasonably clean document image.

Doctr (https://github.com/mindee/doctr) is a new package that uses modern methods for both text detection and recognition. It is pretty new however and I expect will take more time and effort to mature.

https://github.com/PaddlePaddle/PaddleOCR

Personally I‘ve used Tesseract before but the results were underwhelming, so I‘m curious how Paddle OCR performs in comparison.

What kind of accuracy have you reached with this pipeline of Tesseract+LLM? I imagine that there would be a hard limit as to what level the LLM could improve the OCR extract text from Tesseract, since its far from perfect itself.

Haven't seen many people mention it, but have just been using the PaddleOCR library on it's own and has been very good for me. Often achieving better quality/accuracy than some of the best V-LLM's, and generally much better quality than other open-source OCR models I've tried like Tesseract for example.

https://github.com/PaddlePaddle/PaddleOCR/blob/main/README_e...

https://huggingface.co/spaces/echo840/ocrbench-leaderboard

I tried https://github.com/PaddlePaddle/PaddleOCR for my own use case (scanline images of parcel labels) and it beat Tesseract by an order of magnitude.

(Tesseract managed to get 3 fields out of a damaged label, while PaddleOCR found 35, some of them barely readable even for a human taking time to decypher them)

This is where PaddleOCR comes in—it enhances the vision capabilities of the LLM by providing precise OCR text, helping the model focus on exactly what needs to be extracted.

Was this by any chance Paddle OCR https://github.com/PaddlePaddle/PaddleOCR

Budget Constraints: For users with limited budgets, open-source options like Tesseract OCR or PaddleOCR provide good solutions that can be customized to meet specific business needs. Additionally, consider Klippa or API4AI OCR for affordable yet reliable OCR services that work out-of-the-box.

Paddle OCR

If you want to run locally you can look into this https://github.com/PaddlePaddle/PaddleOCR

https://andrejusb.blogspot.com/2024/03/optimizing-receipt-pr...

But I suggest that you just skip that and use gpt-4o. They aren't actually going to steal your data.

Sort through it to find anything with a credit card number or anything ahead time.

Or you could look into InternVL..

Or a combination of PaddleOCR first and then use a strong LLM via API, like gpt-4o or llama3 70b via together.ai

If you truly must do it locally, then if you have two 3090s or 4090s it might work out. Otherwise it the LLMs may not be smart enough to give good results.

Leaving out the details of your hardware makes it impossible to give good advice about running locally. Other than, it's not really necessary.

PyTesseract Module [ Github ] EasyOCR Module [ Github ] PaddlePaddle OCR [ Github ]