Top 14 pre-training Open-Source Projects

• LLMSurvey

  3 8,825 7.9 Python

  The official GitHub page for the survey paper "A Survey of Large Language Models".

  

Project mention: Ask HN: Textbook Regarding LLMs | news.ycombinator.com | 2024-03-23

Here’s another one - it’s older but has some interesting charts and graphs.

https://arxiv.org/abs/2303.18223

• Oscar

  1 1,030 4.0 Python

  Oscar and VinVL

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Awesome-CLIP

  2 1,019 0.0

  Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).

• TencentPretrain

  1 981 7.6 Python

  Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo

  

• Awesome-TimeSeries-SpatioTemporal-LM-LLM

  1 742 7.1

  A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.

Project mention: Awesome-TimeSeries-AIOps-LM-LLM: NEW Data - star count:169.0 | /r/algoprojects | 2023-10-21

• XPretrain

  1 438 1.0 Python

  Multi-modality pre-training

  

Project mention: CVPR 2024 Datasets and Benchmarks - Part 1: Datasets | dev.to | 2024-04-23

It was created by curating 3.8 million high-resolution videos from the publicly available HD-VILA-100M dataset. The dataset is created following these three steps:

• uni2ts

  2 416 7.6 Jupyter Notebook

  Unified Training of Universal Time Series Forecasting Transformers

  

Project mention: Moirai: A Time Series Foundation Model for Universal Forecasting | news.ycombinator.com | 2024-03-25

Code is available! https://github.com/SalesforceAIResearch/uni2ts

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• conceptual-12m

  1 305 0.0

  Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.

  

• STEP

  1 305 4.0 Python

  Code for our SIGKDD'22 paper Pre-training-Enhanced Spatial-Temporal Graph Neural Network For Multivariate Time Series Forecasting. (by zezhishao)

• GearNet

  1 242 4.4 Python

  GearNet and Geometric Pretraining Methods for Protein Structure Representation Learning, ICLR'2023 (https://arxiv.org/abs/2203.06125)

  

• Bamboo

  3 161 4.1 Python

  Bamboo: 4 times larger than ImageNet; 2 time larger than Object365; Built by active learning. (by ZhangYuanhan-AI)

• AMRBART

  1 89 2.8 Python

  Code for our paper "Graph Pre-training for AMR Parsing and Generation" in ACL2022

• TensorFlowTTS-ts

  1 6 5.5 TypeScript

  This project implements TensorflowTTS in Tensorflow.js using Typescript, enabling real-time text-to-speech in the browser. With pre-trained model for English language, you can generate high-quality speech from text input.

• tsdae

  1 3 5.1 Python

  Tranformer-based Denoising AutoEncoder for Sentence Transformers Unsupervised pre-training.

Project mention: Tranformer-based Denoising AutoEncoder for ST Unsupervised pre-training | news.ycombinator.com | 2024-02-04

A new PyPI package for training sentence embedding models in just 2 lines.

The acquisition of sentence embeddings often necessitates a substantial volume of labeled data. However, in many cases and fields, labeled data is rarely accessible, and the procurement of such data is costly. In this project, we employ an unsupervised process grounded in pre-trained Transformers-based Sequential Denoising Auto-Encoder (TSDAE), introduced by the Ubiquitous Knowledge Processing Lab of Darmstadt, which can realize a performance level reaching 93.1% of in-domain supervised methodologies.

The TSDAE schema comprises two components: an encoder and a decoder. Throughout the training process, TSDAE translates tainted sentences into uniform-sized vectors, necessitating the decoder to reconstruct the original sentences utilizing this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embeddings from the encoder. Subsequently, during inference, the encoder is solely utilized to form sentence embeddings.

GitHub : https://github.com/louisbrulenaudet/tsdae

Installation :

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

pre-training related posts

• Awesome CLIP

  1 project | news.ycombinator.com | 22 Feb 2023

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com