unstructured
llm
unstructured | llm | |
---|---|---|
12 | 28 | |
7,108 | 3,268 | |
9.7% | - | |
9.8 | 9.3 | |
6 days ago | 12 days ago | |
HTML | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
unstructured
-
LlamaCloud and LlamaParse
Be careful with unstructured:
https://github.com/Unstructured-IO/unstructured/blob/d11c70c...
from: https://github.com/open-webui/open-webui/issues/687
- FLaNK 15 Jan 2024
-
Bash One-Liners for LLMs
I’ve been looking at this
https://freeling-user-manual.readthedocs.io/en/v4.2/modules/...
at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly.
There is also
https://github.com/Unstructured-IO/unstructured
But I don’t like it, can’t explain why yet.
My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text.
BRIO/BERT summarization could also have a role of some type.
Those are my ideas so far.
- Unstructured – OSS libraries and APIs to build custom preprocessing pipelines
-
More intelligent Pdf parsers
Unstructured is the best one I’ve used so far: https://www.unstructured.io
- Help extracting data from multiple PDF's
- Pre-processing text documents such as PDFs, HTML and Word Documents for LLMs
-
Using ChatGPT to read multiple PDFs and create writing using them as sources
https://www.unstructured.io/ can parse PDFs, then you can feed all of them to Claude, which has a 100k context window.
-
How can I convert restaurant’s traditional menu in pdf file to well structured list of menu items with prices in Excel file? Thank you
If the copy & pase method does not work: One approach is to use the functionality of Unstructured to parse the PDF. If need be, it can do OCR on the PDF too if you have Detectron2 installed. After conversion you would still have to save it as an excel file though.
-
PDF GPT allows you to chat with the contents of your PDF file
I would check out https://github.com/Unstructured-IO/unstructured (what lang chain uses) or https://github.com/axa-group/Parsr (probably what unstructured copied to get their startup off the ground lol)
llm
-
Show HN: PDF to Podcast – Convert Any PDF into a Podcast Episode
I run MacWhisper on my laptop, and often dump podcast MP3s into it, extract the Whisper transcript and then feed that through a long context model like Claude 3 Haiku/Opus or Gemini Pro 1.5/Gemini Flash using my https://llm.datasette.io/ tool to answer questions against that transcript.
- Access LLMs from the Command Line
-
iTerm2 and AI Hype Overload
Access LLMs from the command line: https://github.com/simonw/llm
-
Show HN: Interactive Graph by LLM (GPT-4o)
- *Description*: The `llm` command-line tool leverages large language models, such as OpenAI's GPT-3, to make it easier to incorporate AI functionalities into your command-line tasks. You can use it to generate text, answer questions, and assist with coding or other language-based tasks directly from your terminal.
- **Link**: [llm on GitHub](https://github.com/simonw/llm)
-
GPT-4o
Slight off-topic, but I noticed you've updated your llm CLI app to work with the 4o model (plus bunch of other APIs through plugins). Kudos for working extremely fast. I'm really grateful for your tool; I tried many others, but for some reason none clicked as much as your.
Link in case other readers are curious: https://llm.datasette.io
- FLaNK AI-April 22, 2024
-
Show HN: I made a tool to clean and convert any webpage to Markdown
That's a great use case, you might be able to do this if you've got a copy and paste on the command line with
https://github.com/simonw/llm
In between. An alias like pdfwtf translating to "paste | llm command | copy"
-
Command R+: A Scalable LLM Built for Business
I added support for this model to my LLM CLI tool via a new plugin: https://github.com/simonw/llm-command-r
So now you can do this:
pipx install llm
-
The Next Generation of Claude (Claude 3)
If you're willing to use the CLI, Simon Willison's llm library[0] should do the trick.
[0] https://github.com/simonw/llm
- Show HN: I made an app to use local AI as daily driver
What are some alternatives?
llmsherpa - Developer APIs to Accelerate LLM Projects
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
Parsr - Transforms PDF, Documents and Images into Enriched Structured Data
langroid - Harness LLMs with Multi-Agent Programming
ragflow - RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
exllama - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
awesome-document-understanding - A curated list of resources for Document Understanding (DU) topic
multi-gpt - A Clojure interface into the GPT API with advanced tools like conversational memory, task management, and more
pdfGPT - PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. The most effective open source solution to turn your pdf files in a chatbot!
jehuty - Fluent API to interact with chat based GPT model
llama_parse - Parse files for optimal RAG
llm-replicate - LLM plugin for models hosted on Replicate