DeepSpeed VS Pytorch

Filed a discussion item on the deepspeed project: https://github.com/microsoft/DeepSpeed/discussions/3531

Solution: I don't know; this is where I am stuck. https://github.com/microsoft/DeepSpeed/issues/1037 suggests that I just need to 'apt install libaio-dev', but I've done that and it doesn't help.

There could be some spectrum of this expertise. For instance, https://github.com/NVIDIA/FasterTransformer, https://github.com/microsoft/DeepSpeed

DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat)

I'm following this closely, together with other efforts like GPTQ Quantization and Microsoft's DeepSpeed, all of which are bringing down the hardware requirements of these advanced AI models.

- https://github.com/microsoft/DeepSpeed

Anything that could bring this to a 10GB 3080 or 24GB 3090 without 60s/it per token?

The original Torch library (the predecessor to PyTorch) was a Lua library. A lot of early 2010s NN research was done in Lua.

This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2]

[1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b...

[2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...

const transformLayer = nn.Einsum((Batch, Seq, Emb),(Emb)->(Batch, Seq))

const tensorB: Tensor[Emb2] = t.randn([20])

const transformedOutput = transformLayer(tensorA, tensorB) // type error: Emb2 does not match Emb

```

[0]: https://github.com/pytorch/pytorch/issues/26889

Click to Learn more...

It's the same with pytorch 2.0. I think it's related to this issue: https://github.com/pytorch/pytorch/issues/90920

Install pytorch with conda. See pytorch.org for details. conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

I tend to be most impressed by tools and libraries. The stuff that has most impressed me in my time in ML is stuff like pytorch and Stan, tools that allow expression of a wide variety of statistical (and ML, DL models, if you believe there's a distinction) models and inference from those models. These are the things that have had the largest effect in my own work, not in the sense of just using these tools, but learning from their design and emulating what makes them successful.

>I work for AWS; these are the definitions we use, more or less.

you're just being asinine - we're literally talking about binary code that's never seen by anyone that doesn't compile from source and goes digging around in the build dir - how could you possibly call that code "transparent" in any sense of the word? are blob drivers also transparent according to these "AWS" definitions?

>I was specifically referring to the computation graph of the model that is used by autograd.

it's literally right there in bolded text on the first page of the original paper (the 2017 neurips paper):

>Immediate, eager execution. An eager framework runs tensor computations as it encounters them; it avoids ever materializing a “forward graph”, recording only what is necessary to differentiate the computation

autograd has absolutely nothing to do with the graph - autograd is literally 10s of thousands of lines of generated, templatized, code that connects edges one op at a time. you can argue with me all you want or you can just go to repo tip and see for yourself https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/t...

PyTorch hasn't even been released for 5.5 and it's been a week. Someone finally opened an issue asking them to look at it.

Home-page: https://pytorch.org/