[D] Mixed Precision Training: Difference between BF16 and FP16

• pytorch-lightning

  9 26,883 9.9 Python

  Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.

  

For the A100 GPU, theoretical performance is the same for FP16/BF16 and both rely on the same number of bits, meaning memory should be the same. However since it's quite newly added to PyTorch, performance seems to still be dependent on underlying operators used (pytorch lightning debugging in progress here).

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project