unilm VS donut

+1 On this, the real proof would have been testing both models side-by-side.

It seems that it may be published on GitHub [1] according to HuggingFace [2].

[1] https://github.com/microsoft/unilm/tree/master/bitnet

[2] https://huggingface.co/papers/2402.17764

e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69...

they actually have custom prompts for each dataset being tested.

Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task?

IMO e5-mistral is overfit to MTEB

Layout LM v1, v2 and v3 models [ Github ] DocBERT [ Github ]

The repository is available here.

It is missing kosmos-2. I remember its image captioning was(demo currently down) really good and it's almost as fast as llava and lavin.

Should be this: https://github.com/microsoft/unilm/

This is from Microsoft Research (Asia). https://aka.ms/GeneralAI

maybe this is better? https://github.com/clovaai/donut

I'm not sure

Yup! But I'm still exploring options. (any recommendations would be welcomed!) Here are some candidates I'm considering:

- https://github.com/mindee/doctr

- https://github.com/open-mmlab/mmocr

- https://github.com/PaddlePaddle/PaddleOCR (honestly I don't know Mandarin so I'm a bit stuck)

- https://github.com/clovaai/donut - While it's primarily an "OCR-free document understanding transformer," I think it's worth experimenting with. Think I can sort this out by letting the LLM reason through it multiple times (although this will impact performance)

- yesterday got a suggestion to consider https://github.com/kakaobrain/pororo - I don't think development is still active but the results are pretty great on Korean text

I am also interested in experimenting with something like DONUT (https://github.com/clovaai/donut) but I have never seen anything on what the VRAM expectations are for something like this. Does anyone know also if there are any newer better models than this for document parsing as well? Or what the VRAM requirements for something like this tend to be?

Here are a few useful resources you could start with: [Pix2Struct by Google Research](https://github.com/google-research/pix2struct) might be a valuable tool, although it will most likely need some fine-tuning to fit your specifics. You can also find some fine-tuned models on HuggingFace by searching 'pix2struct'. Another option worth considering is [DonutI](https://github.com/clovaai/donut). Like Pix2Struct, fine-tuning likely needed to meet your requirements. Tesseract OCR is another alternative, particularly for handling text. It's primarily designed for pages of text, think books, but with some tweaking and specific flags, it can process tables as well as text chunks in regions of a screenshot. Bit too much tweaking for my taste. As I'm also in search of OCR tools for UI and chart screenshots, so share if you find something else.