unstructured VS awesome-document-understanding

Be careful with unstructured:

https://github.com/Unstructured-IO/unstructured/blob/d11c70c...

from: https://github.com/open-webui/open-webui/issues/687

I’ve been looking at this

https://freeling-user-manual.readthedocs.io/en/v4.2/modules/...

at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly.

There is also

https://github.com/Unstructured-IO/unstructured

But I don’t like it, can’t explain why yet.

My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text.

BRIO/BERT summarization could also have a role of some type.

Those are my ideas so far.

Unstructured is the best one I’ve used so far: https://www.unstructured.io

https://www.unstructured.io/ can parse PDFs, then you can feed all of them to Claude, which has a 100k context window.

If the copy & pase method does not work: One approach is to use the functionality of Unstructured to parse the PDF. If need be, it can do OCR on the PDF too if you have Detectron2 installed. After conversion you would still have to save it as an excel file though.

I would check out https://github.com/Unstructured-IO/unstructured (what lang chain uses) or https://github.com/axa-group/Parsr (probably what unstructured copied to get their startup off the ground lol)

I have worked on several Document Understanding (DU) projects for my company during the last year. We've mainly used UiPath and Google's DocumentAI.

While trying to find a specific project I recalled, I encountered this list of projects which might be of interest: https://github.com/tstanislawek/awesome-document-understandi...

The project I had in mind was similar to this one but I can't remember the name currently: https://github.com/tabulapdf/tabula

However, if you're looking for a ML-based, invoice-specific project looks like the other comment to your reply might be more useful.

Check out this repository for inspiration: https://github.com/tstanislawek/awesome-document-understanding

In the last few years, I spent a lot of time working on automate business processes of big companies and seeing rising interest in DU topics (especially from Key Information Extraction field). Therefore, I create a list https://github.com/tstanislawek/awesome-document-understanding of resources to make easier to track all the papers out there which are relevant to this topic.