Ask HN: Can you crowdfund the compute for GPT?

• petals

  98 8,684 8.3 Python

  🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

  

Yes: https://github.com/bigscience-workshop/petals

• ai_and_memory_wall

  1 183 3.9

  AI and Memory Wall

Depends on which version you're running. According to https://github.com/amirgholami/ai_and_memory_wall

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• metaseq

  53 6,386 6.2 Python

  Repo for external large-scale work

  

https://github.com/facebookresearch/metaseq/blob/main/projec... [PDF]

It's a 114 page document detailing the months of full-time work for multiple engineers that went into training a 175B language model at meta AI.

• GLM-130B

  19 7,610 4.8 Python

  GLM-130B: An Open Bilingual Pre-Trained Model (ICLR 2023)

  

https://github.com/THUDM/GLM-130B might be a useful place to look

• hivemind

  40 1,837 5.4 Python

  Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

  

Probably the biggest recent result: https://arxiv.org/abs/2209.04836

See also: https://github.com/learning-at-home/hivemind

and more to OP's incentive structure: https://docs.bittensor.com/

Latter two intend to beat latency with Mixture-of-Expert models (MOEs). If the results of the former hold, it shows that with a simple algorithmic transformation you can merge two independently trained models in weight-space and have performance functionally equivalent to a model trained monolithically.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project