gpt4-pdf-chatbot-langchain VS semantic-search-tweets

I am playing around with this github project, which takes a user question as input and immediately runs a vector search on it to find relevant storied information before delivering an answer.

I am using this https://github.com/mayooear/gpt4-pdf-chatbot-langchain as a reference to ingest PDFs into pinecone and chat with a document, but my results aren’t good. Since it’s looking for related documents, there’s no good relation to the meta question: “What questions were asked in this interview?”

Anyone can already do this locally with their own API keys for free, with no technical knowledge by cloning a github repo (e.g. https://github.com/mayooear/gpt4-pdf-chatbot-langchain - this one can also chat with multiple pdfs which is much better). Even with gpt-4, I just don't find the responses useful usually. I find the model doesn't do great with scientific stuff aside from asking very basic things. Might have to wait for gpt-5.

https://github.com/mayooear/gpt4-pdf-chatbot-langchain this repo uses gpt-3.5/4 which uses OpenAI API. Is there any work donw with free/open-source LLMs

Follow this guide https://github.com/mayooear/gpt4-pdf-chatbot-langchain

I am using this one on windows 10. Took 2 evenings to set up: https://github.com/mayooear/gpt4-pdf-chatbot-langchain

Thanks for your take on the subject. I agree that starting from scratch would be too much. I think my post above might be misleading in regard to training. I wouldn't suggest to start from scratch but to provide additional data to a pretrained AI. But you can use GPT-4 (through API) in combination with pinecone to provide data. Here is a project, where someone implemented this to work with large PDF files. I don't think it would be too hard, to start from there and adapt the project to the requirements of OP. Obviously this would require paid for API keys. LLama could be also a good starting point, with a lot of resources available.

I'm currently developing a chatbot application that interacts with PDF documents using GPT API, Langchain, and a Pinecone vector database. The project is built on this repository: mayooear/gpt4-pdf-chatbot-langchain.

It's in the repo:

You first create embeddings. What is this? It's an n-dimensional vector space with your tweets 'embedded' in that space. Each word is an n-dimensional vector in this space. The vectorization is supposed to maintain 'semantic distance'. Basically, if two words are very close in meaning or related (by say frequently appearing next to each other in corpus) they should be 'close' in some of those n-dimensions as well. The result at the end is the '.bin' file, the 'semantic model' of your corpus.

https://github.com/dbasch/semantic-search-tweets/blob/main/e...

For semantic search, you run the same embedding algorithm against the query and take the resultant vectors and do similarity search via matrix ops, resulting in a set of results, with probabilities. These point back to the original source, here the tweets, and you just print the tweet(s) that you select from that result set.

https://github.com/dbasch/semantic-search-tweets/blob/main/s...

Experts can chime in here but there are knobs such as 'batch size' and the functions you use to index. (cosine was used here.)

So the various performance dimensions of the process should also be clear. There is a fixed cost of making the embeddings of your data. There is a per-op embedding of your query, and then running the similarity algorithm to find the result set.