Ask HN: What are you using to parse PDFs for RAG?

• marker

  17 16,192 8.7 Python

  Convert PDF to markdown quickly with high accuracy

  Hi, I'm the author of marker - https://github.com/VikParuchuri/marker - from my testing, marker handles almost all the issues you mentioned. The biggest issue (that I'm working on fixing right now) is formatting tables properly.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• pdf2htmlEX

  4 3,653 3.4 HTML

  Convert PDF to HTML without losing text or format. (by pdf2htmlEX)

  

  Previously have used https://github.com/pdf2htmlEX/pdf2htmlEX to convert PDF to HTML at scale, could potentially try and parse the output html to markdown as second stage.

• PyMuPDF

  7 4,986 9.8 Python

  PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.

  

• RAG

  1 255 8.7 Python

  RAG (Retrieval-Augmented Generation) Chatbot Examples Using PyMuPDF (by pymupdf)

  

• pandoc

  425 33,852 9.8 Haskell

  Universal markup converter

• nougat

  16 8,680 7.5 Python

  Implementation of Nougat Neural Optical Understanding for Academic Documents

  

• AI-in-a-Box

  1 502 9.2 Jupyter Notebook

  

  https://github.com/Azure/AI-in-a-Box/tree/main/ai-services/d...

• InfluxDB

  www.influxdata.com featured

  Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

  

• open-parse

  4 2,325 9.0 Python

  Improved file parsing for LLM’s

• document-ai-samples

  6 229 8.5 Jupyter Notebook

  Sample applications and demos for Document AI, the end-to-end document processing platform on Google Cloud

  

• layout-parser

  7 4,763 0.0 Python

  A Unified Toolkit for Deep Learning Based Document Image Analysis

  

• PDF-Extract-Kit

  1 4,529 8.6 Python

  A Comprehensive Toolkit for High-Quality PDF Content Extraction

  

• PaddleOCR

  65 42,259 9.1 Python

  Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)

  

• WDoc

  3 59 10.0 Python

  Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, scalable, under developpement

  For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser then use heuristics the keep the best one. The code is at https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

  And the heurstics are partly based on using fasttext to detecr languages : https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

  It's probably crap for tables but I don't want to rely on external parsers.

Suggest a related project