datasets VS sentence-transformers

https://github.com/huggingface/datasets

https://github.com/huggingface/transformers

And then pass that my_code.json as the dataset name.

[1] https://github.com/huggingface/datasets

This article shows how txtai can index and search with Hugging Face's Datasets library. Datasets opens access to a large and growing list of publicly available datasets. Datasets has functionality to select, transform and filter data stored in each dataset.

huggingface/datasets (Python): 🤗 Fast, efficient, open-access datasets and evaluation metrics for Natural Language Processing and more in PyTorch, TensorFlow, NumPy and Pandas

Are you using one of these? https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/ms_marco/train_bi-encoder_mnrl.py https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/ms_marco/train_bi-encoder_mnrl.py

Break up transcript into shorter segments and convert segments to a 768 vector array. Use a process known as embedding using our second ML model, UKP Labs BERT’s sentence transformer model.

Not sure what kind of texts you have, but these models have a max sequence limit of 512 (approx 350 words or so). If you're texts are longer than that, consider splitting them up into chunks or creating a summary and taking an embedding of that. Some clustering algorithm may be the way to go here. Here's a bunch of examples. I use agglomerative for my use case.

Sounds good. In terms of the theory behind the models, transformers and sentence-transformers are best projects to take a look at.

We made this using pretty recent advances in web-data pretrained models like Detic and LSeg for detection, CLIP for visual queries, and Sentence BERT for semantic queries. Our "database" is really a neural field (Instant NGP) that maps from 3D coordinates to a high dimensional embedding vector in the same representation space as CLIP and SBERT.

Did you use venv, or did you work around this another way?

A number of excellent topic modeling libraries exist in Python today. BERTopic and Top2Vec are two of the most popular. Both use sentence-transformers to encode data into vectors, UMAP for dimensionality reduction and HDBSCAN to cluster nodes.

The ONNX pipeline also supports exporting sentence embeddings models trained with the sentence-transformers package.

sentence-transformers recently added support for the OpenAI CLIP model. This model embeds text and images into the same space, enabling image similarity search. txtai can directly utilize these models through sentence-transformers. Check out the sentence-transformers link above for additional examples on how to use this model.