Ask HN: What are you using to parse PDFs for RAG?

1. marker

   1 30 24,879 9.8 Python

   Convert PDF to markdown + JSON quickly with high accuracy

   

   Hi, I'm the author of marker - https://github.com/VikParuchuri/marker - from my testing, marker handles almost all the issues you mentioned. The biggest issue (that I'm working on fixing right now) is formatting tables properly.

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. pdf2htmlEX

   2 4 4,944 3.4 HTML

   Convert PDF to HTML without losing text or format. (by pdf2htmlEX)

   

   Previously have used https://github.com/pdf2htmlEX/pdf2htmlEX to convert PDF to HTML at scale, could potentially try and parse the output html to markdown as second stage.

4. PyMuPDF

   3 8 7,124 9.7 Python

   PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.

   

5. RAG

   4 1 902 7.9 Python

   RAG (Retrieval-Augmented Generation) Chatbot Examples Using PyMuPDF (by pymupdf)

   

6. pandoc

   5 444 37,278 9.8 Haskell

   Universal markup converter

   

7. nougat

   6 19 9,436 2.4 Python

   Implementation of Nougat Neural Optical Understanding for Academic Documents

   

8. AI-in-a-Box

   7 1 572 8.4 Jupyter Notebook

   AI-in-a-Box leverages the expertise of Microsoft across the globe to develop and provide AI and ML solutions to the technical community. Our intent is to present a curated collection of solution accelerators that can help engineers establish their AI/ML environments and solutions rapidly and with minimal friction.

   

   https://github.com/Azure/AI-in-a-Box/tree/main/ai-services/d...

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

10. open-parse

    8 4 2,944 8.9 Python

    Improved file parsing for LLM’s

    

11. document-ai-samples

    9 7 272 6.6 Jupyter Notebook

    Sample applications and demos for Document AI, the end-to-end document processing platform on Google Cloud

    

12. layout-parser

    10 7 5,233 0.0 Python

    A Unified Toolkit for Deep Learning Based Document Image Analysis

    

13. PDF-Extract-Kit

    11 2 7,548 8.9 Python

    A Comprehensive Toolkit for High-Quality PDF Content Extraction

    

14. PaddleOCR

    12 69 49,024 9.4 Python

    Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)

    

15. wdoc

    13 7 451 9.9 Python

    Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, etc

    

    For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser then use heuristics the keep the best one. The code is at https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

    And the heurstics are partly based on using fasttext to detecr languages : https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

    It's probably crap for tables but I don't want to rely on external parsers.

Suggest a related project