tldw VS wdoc

Hey yes, I’m building exactly that.

https://github.com/rmusser01/tldw

I first built a POC in gradio and am now rebuilding it as a FastAPI app. The media processing endpoints work but I’m still tweaking media ingestion to allow for syncing to clients(idea is to allow for client-first design).

You could try my app https://github.com/rmusser01/tldw

Supports arbitrary length videos and also lets you choose what LLM API to use.

Not the person you asked, but it's dependent on what you're trying to chunk. I've written a standalone chunking library for an app I'm building: https://github.com/rmusser01/tldw/blob/main/App_Function_Lib...

It's setup so that you can perform whatever type of chunking you might prefer.

As someone who's built something like it in their free time as a hobby project ( https://github.com/rmusser01/tldw), could I ask what would make it a professional product vs something an intern came up with? Looking for insights I could possibly apply/learn from to implement in my own project.

One of my goals with my project I ended up taking on was to match/exceed NotebookLMs feature set, to ensure that an open source version would be available to people for free, with ownership of their data.

Hey I’m working on exactly this: https://github.com/rmusser01/tldw

It’s still a work in progress but my goal is to make an open source solution for exactly what you describe to help people. (Starting with myself :p)

I recently built a semantic batching function for my RAG system [wdoc](https://github.com/thiswillbeyourgithub/wdoc/) that might be interesting to others. The system splits a corpus into chunks, finds relevant ones via embeddings, and answers questions for each chunk in parallel before aggregating the answers.

To optimize performance and reduce LLM distraction, instead of aggregating answers two by two, it does batched aggregation. The key innovation is in the batching order - I implemented a [semantic_batching function](https://github.com/thiswillbeyourgithub/wdoc/blob/18bc52128f...) that uses hierarchical clustering on the embeddings and orders texts by leaf order.

The implementation was straightforward, runs very fast and produces great results. The function is designed to be usable as a standalone tool for others to experiment with.

I've made wdoc just for that: https://github.com/thiswillbeyourgithub/WDoc

I am a medical student with thousands of pdfs, various anki databases, video conferences, audio recordings, markdown notes etc. It can query into all of them and return extremely high quality output with sources to each original document.

It's still in alpha though and there's only 0.5 user beside me that I know of so there are bugs that have yet to be found!

Here's a highlight (edit: more like an ego dump)

I couldn't keep up with my news so I made the perfect summarizer that goes through the thought process of the author : https://github.com/thiswillbeyourgithub/WDoc

I needed an AI based system that go through my anki cards, but might as well make it able to read dozens of file formats. Now I can put entire medical youtube playlists, conferences, anki databases, hundreds of PDFs and ask a single question across all of them at once .

It's both the same project

For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser then use heuristics the keep the best one. The code is at https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

And the heurstics are partly based on using fasttext to detecr languages : https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2...

It's probably crap for tables but I don't want to rely on external parsers.

Don't hesitate to ask for features!

Here's the link: https://github.com/thiswillbeyourgithub/DocToolsLLM/