1filellm VS gpt-repository-loader

I created something similar, https://github.com/jimmc414/1filellm

It converts papers, repositories, PRs and web docs into one text file for llm ingestion

I created a cli tool that copies a GitHub or local repo into a text file for llm ingestion. It only pulls the filetypes you specify.

https://github.com/jimmc414/1filellm

Nice. I guess we are all thinking the same things. I created something similar today where you can choose the file extensions to pull and handles Jupiter notebooks pulling only text and code. I included a script to strip out superfluous characters, stop words, and converts everything to lowercase.

It also works if you supply a local folder of source files instead of a github repo.

https://github.com/jimmc414/onefilerepo

A good approach is to use a tool like this that turns an entire repo into a single file you can upload to GPT: https://github.com/mpoon/gpt-repository-loader (not my own repo). I find that much cleaner than doing a bunch of copy-and-pasting or uploading files one-by-one.

I have made an AI-bot that knows ALL of NextJS set up on my localhost. I can ask anything and it outputs the answer, context, examples and references. I used https://github.com/mpoon/gpt-repository-loader to create the files and loaded them into https://github.com/dabit3/semantic-search-nextjs-pinecone-langchain-chatgpt , then added OpenAI API to shape the { result } to my wishes. Works decently

You might start with this: https://github.com/mpoon/gpt-repository-loader

You may be interested in this: https://github.com/mpoon/gpt-repository-loader

I've had access to the 32k model for a bit and I've been using this to collect and stuff codebases into the context: https://github.com/mpoon/gpt-repository-loader

It works really well, you can tell it to implement new features or mutate parts of the code and it having the entire (or a lot of) the code in its context really improves the output.

The biggest caveat: shit is expensive! A full 32k token request will run you like $2, if you do dialog back and forth you can rack up quite the bill quickly.

https://github.com/mpoon/gpt-repository-loader here is open source take on part of this concept