Extracting-Training-Data-from-Large-Langauge-Models
A re-implementation of the "Extracting Training Data from Large Language Models" paper by Carlini et al., 2020 (by shreyansh26)
finetune-gpt2xl
Guide: Finetune GPT2-XL (1.5 Billion Parameters) and finetune GPT-NEO (2.7 B) on a single GPU with Huggingface Transformers using DeepSpeed (by Xirider)
| Extracting-Training-Data-from-Large-Langauge-Models | finetune-gpt2xl | |
|---|---|---|
| 1 | 9 | |
| 26 | 422 | |
| - | - | |
| 5.1 | 0.0 | |
| almost 2 years ago | 11 months ago | |
| Python | Python | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Extracting-Training-Data-from-Large-Langauge-Models
Posts with mentions or reviews of Extracting-Training-Data-from-Large-Langauge-Models.
We have used some of these posts to build our list of alternatives
and similar projects.
-
Paper Implementation - Extracting Training Data from Large Language Models
Code - https://github.com/shreyansh26/Extracting-Training-Data-from-Large-Langauge-Models
finetune-gpt2xl
Posts with mentions or reviews of finetune-gpt2xl.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-02-13.
-
Fine-tuning?
git clone the finetuning repo https://github.com/Xirider/finetune-gpt2xl go into the finetuning repo, install the rest of the requirements, pip install -r requirements.txt
- Training text-generating models locally
-
Dataset For GPT Fine-Tuning
I would like to understand a little better how to organize texts for Fine-Tuning, especially for GPT Neo. I plan to use this repo procedure, where is the following notice,
-
How to share the finetuned model
In the code suggested in the video (and in the repo) the flag --fp16 is used. But reading the "DeepSpeed Integration" article it is said that,
- [D] I made a script that does all the work to deploy GPT-NEO on Windows 10. (Please Test)
-
[Project] Estimating fine-tuning cost
Finetuning GPT-NEO 2.7B on Wikitext (180mb) took me about 45 minutes on one preemptible V100 instance on google cloud. It cost 1.30$ per hour and therefore around 1 $. Here are the steps: https://github.com/Xirider/finetune-gpt2xl
-
[P] Guide: Finetune GPT2-XL (1.5 Billion Parameters, the biggest model) on a single 16 GB VRAM V100 Google Cloud instance with Huggingface Transformers using DeepSpeed
Here i explain the setup and commands to get it running: https://github.com/Xirider/finetune-gpt2xl
- Guide: Finetune GPT2-XL (1.5 Billion Parameters, the biggest model) on a single 16 GB VRAM V100 Google Cloud instance with Huggingface Transformers using DeepSpeed
What are some alternatives?
When comparing Extracting-Training-Data-from-Large-Langauge-Models and finetune-gpt2xl you can also consider the following projects:
Neural-Scam-Artist - Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.
detoxify - Trained models & code to predict toxic comments on all 3 Jigsaw Toxic Comment Challenges. Built using ⚡ Pytorch Lightning and 🤗 Transformers. For access to our API, please email us at [email protected].
funcbygpt - tell the function what to do and gpt will write the code