[D] Fine-tuning GPT-J: lessons learned

• Finetune_LLMs

  2 436 8.5 Python

  Repo for fine-tuning Casual LLMs

And this: https://github.com/mallorbc/Finetune_GPTNEO_GPTJ6B

• DeepSpeed

  51 32,550 9.8 Python

  DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

  

Regarding inference, it seems that Deepspeed doesn't support inference for GPT-J, but they are planning to work on it: https://github.com/microsoft/DeepSpeed/issues/1332

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• mesh-transformer-jax

  52 6,213 0.0 Python

  Model parallel transformers in JAX and Haiku

I don't know I was surprised too. The config recommended for TPU in the how-to we used is hard to compare with the one we used for fine-tuning on GPU with Deepspeed. So maybe both are not exactly equal and we're comparing apples to oranges...

Suggest a related project