finetuner VS qdrant

You can try creating your own dataset. Get some audio data that you want, preprocess it, and then create a custom dataset you can use to fine tune. You could use finetuners like these if you want as well.

We derived the dataset by pre-processing the deepfashion dataset using Finetuner. The image label generated by Finetuner is extracted and formatted to produce the text attribute of each product.

You can either use Hugging Face Transformers as they have a lot of pre-trained models that you can customize. Or Finetuners like this one: which is a toolkit for fine-tuning multiple models.

Very recently, a few non-English and multilingual CLIP models have appeared, using various sources of training data. In this article, we’ll evaluate a multilingual CLIP model’s performance in a language other than English, and show how you can improve it even further using Jina AI’s Finetuner.

you can try prompt-based classification or fine-tuning with a Finetuner. Prompts work well for simple tasks but fine-tuning may give better results for complex ones. Althouigh it's going to need more resources, but try both and see what works best for you.

If you've got a set of Q&A pairs for your 60-page lease or medical paper, you could use finetuners to help answer questions about the text. But if you don't have those pairs, fine-tuning might not be good. Try summarizing the doc or extract the info. And if you're hitting the token limit, try using a bigger model or breaking up the text into smaller pieces.

Actually further in applying ML, Finetuner is pretty handy for getting the last mile done which I found useful.

One option I’m considering is using fine tuners like the one from HuggingFace or Jina AI to fine-tune open source models like GPT-J or OPT to improve specific use-cases like code explanation. With the funding that we have, I wouldn’t want to cheap out on fine-tuning and expect something good.

Off the top of my head you can either use Grid Search to test hyperparam combinations, Random Search to randomize hyperparams and Neural search uses ML to optimize hyperparameter tuning. You can use finetuners for this as well.

Back then, I came across some info about a self-supervised sentence embedding system that surpasses Sentence Transformers NLI models, but forgot where it was. You could use Jina’s Finetuner. It lets you boost your pre-trained models' performance, making them ready for production without having to spend a lot of time labeling or buying expensive hardware.

Qdrant | REMOTE | Full-time | https://qdrant.tech | https://github.com/qdrant/qdrant

Qdrant is a leading open-source Vector Database provider.

We are Looking for Technical Writer, Integrations Engineer, Database Tester, Developer Advocate(s).

All jobs https://qdrant.join.com

Awesome work!

At Qdrant we do this at scale. Store billions of vectors in a cluster of any size. Also in Rust which turned out to be an amazing choice, and fully open source. It uses various features to keep things performant, such as vectorization (multiple arches), quantization (form of compression) and more.

https://github.com/qdrant/qdrant

It depends on your requirements, for a simple hybrid-search solution, elastic, etc. should be enough. With growing data amount and if working not only with text embeddings, you should try out a dedicated solution, like Qdrant. https://github.com/qdrant/qdrant

Made with Qdrant Vector Database https://github.com/qdrant/qdrant

As an open-source and self-hosted solution, developers can deploy their own version of the plugin and register it with ChatGPT. The plugin leverages OpenAI embeddings and allows developers to choose a vector database (Milvus, Pinecone, Qdrant, Redis, Weaviate or Zilliz) for indexing and searching documents. Information sources can be synchronized with the database using webhooks.

Qdrant

Another example is Bloop, it is a code search engine built on top of tantivy and qdrant.