doctr VS tesserocr

checkout https://github.com/mindee/doctr or https://github.com/VikParuchuri/surya for something practical

multimodal llm would of course blow it all out the water, so some llama3-like model is probably SOTA in terms of what you can run yourself. something like https://huggingface.co/blog/idefics2

https://github.com/mindee/doctr/issues/1049

I am looking for something this polished and reliable for handwriting, does anyone have any pointers? I want to integrate it in a workflow with my eink tablet I take notes on. A few years ago, I tried various models, but they performed poorly (around 80% accuracy) on my handwriting, which I can read almost 90% of the time.

Yup! But I'm still exploring options. (any recommendations would be welcomed!) Here are some candidates I'm considering:

- https://github.com/mindee/doctr

- https://github.com/open-mmlab/mmocr

- https://github.com/PaddlePaddle/PaddleOCR (honestly I don't know Mandarin so I'm a bit stuck)

- https://github.com/clovaai/donut - While it's primarily an "OCR-free document understanding transformer," I think it's worth experimenting with. Think I can sort this out by letting the LLM reason through it multiple times (although this will impact performance)

- yesterday got a suggestion to consider https://github.com/kakaobrain/pororo - I don't think development is still active but the results are pretty great on Korean text

EasyOCR is a popular project if you are in an environment where you can use run Python and PyTorch (https://github.com/JaidedAI/EasyOCR). Other open source projects of note are PaddleOCR (https://github.com/PaddlePaddle/PaddleOCR) and docTR (https://github.com/mindee/doctr).

Last I checked I saw a grocery bill example using https://github.com/mindee/doctr and was fairly accurate. Bear in mind that was last year, hopefully it got even better or there are other libraries

For its OCR service, Cape uses the excellent Python docTR library. Some of the critical benefits of docTR are its ease of use, flexibility, and matching state-of-the-art performance. The OCR model consists of two steps: text detection and text recognition. Cape uses a pre-trained DB Resnet50 architecture for detection, and for recognition, it uses a MobileNetV3 Small architecture. To learn more about the level of OCR accuracy you can expect for your document, you can consult these benchmarks provided by docTR. As you will see, model performance is very competitive compared to other commercial services.

There's also DocTR which can do text detection and extraction out of the box.

It's command line driven but can display the detected text as an overlay of the document.

https://github.com/mindee/doctr

If you want to OCR a document image, modern versions of Tesseract can work well. If you last used it a few years ago, the recognition has improved since due to a new text recognition algorithm that uses modern (deep learning) techniques. Browser demo using a modern version: https://robertknight.github.io/tesseract-wasm/.

OCR processing typically consist of two major steps: detecting/locating words or lines of text on the page, and recognizing lines of text.

Tesseract's text recognition uses modern methods, but the text detection phase is still based on classical methods involving a lot of heuristics, and you may need to experiment with various configuration variables to get the best results. As a result it can fail to detect text if you present it with something other than a reasonably clean document image.

Doctr (https://github.com/mindee/doctr) is a new package that uses modern methods for both text detection and recognition. It is pretty new however and I expect will take more time and effort to mature.

Did you read the instructions for windows? https://github.com/sirfz/tesserocr

If you’re using tesserocr then you can use OpenCV images directly, so you can just extract the relevant image rows (e.g. query_image = main_image[prev_line:this_line]) and process then without needing to save each image.

This tutorial is a good start towards getting the data from an image of a form with a known structure. I’d personally recommend using tesserocr (actual library binding, more efficient, more functionality) instead of pytesseract (requires images to be saved before processing, uses command-line options in a subprocess instead of binding to the library), but both should work (that tutorial uses pytesseract, which is also what u/Iceberg_Bart_Simpson linked to).

Side note, but I’d suggest using tesserocr over pytesseract. It’s an actual binding to the tesseract library, so comes with numerous efficiency and interface benefits, and can operate on OpenCV images directly (whereas pytesseract saves them to disk first).

If you’re after speed I’d recommend mss for screenshots/recording, and tesserocr instead of pytesseract (note in particular the OpenCV option.

tesserocr is an actual binding to the tesseract library, and is better in practically every way than pytesseract (more efficient, more options for usage, doesn’t require saving images to disk before they can be processed, and more).

If you have an electronically created pdf (not scanned) and you’re just wanting to run OCR on embedded images then you’ll want a pdf library that can extract the figure images for you, and then you can use tesserocr to run OCR on those images.

If you want a suggestion, use tesserocr instead of Pytesseract. It’s an actual binding to the tesseract library (Python talks to it directly, instead of calling a program as a subprocess), which means it runs more efficiently, you can process multiple images sequentially with the same OCR engine (pytesseract has to start a process and a new engine for every image that gets processed), you get access to more functionality options, and a bunch of other beneficial stuff. If you’re doing preprocessing with OpenCV it’s even possible to pass those arrays directly to tesseract in memory, whereas Pytesseract requires that you save each image to a file before it can process it.

As a somewhat unrelated side note, I’d strongly suggest using tesserocr instead of pytesseract, and even more so if you’re working with opencv as well. It’s a true library binding which means it’s more efficient, you have more functionality available to you, you can process multiple images with the same Tesseract engine, and you can process opencv images directly (compared to pytesseract which saves them as a file first and then calls the tesseract CLI as a subprocess).

In Python the library PyTesseract constructs a command to run and calls Tesseract via the command-line as a subprocess, which is inefficient if you have more than one image to process, because it has to reinitialize the OCR engine for every image. tesserocr is a different library which came around a bit later, which is a direct binding to the Tesseract library, so you can initialise the engine once and process several images with it, and for images that are stored in memory (e.g. OpenCV arrays that you’ve done some processing on) you can process them directly instead of having to save them as individual files (which PyTesseract requires).