tinygrad VS Polly

Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]?

[0] https://github.com/geohot/tinygrad

at first glance i thought may its like tinygrad. but looks has many ops than that tiny grad but most maps to underlying hardware provided ops?

i wonder how well tinygrad's apporach will work out, ops fusion sounds easy, just a walk a graph, pattern match it and lower to hardware provided ops?

Anyway if anyone wants to understand the philosophy behind tinygrad, this file is great start https://github.com/geohot/tinygrad/blob/master/docs/abstract...

There are currently at least 3 ways to run llama on m1 with GPU acceleration. - mlc-llm (pre-built, only 1 model has been ported) - tinygrad (very memory efficient, not that easy to integrate into other projects) - llama-mps (original llama codebase + llama adapter support)

Hopefully we will see now full support with AMD hardware on https://github.com/geohot/tinygrad. You can read more about it on https://tinygrad.org/

tinygrad

https://github.com/geohot/tinygrad/tree/master/accel/ane

But I have not tested it on Linux since Asahi has not yet added support.

llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) without quantization.

In our applications, we should wrap all requests to remote services in code that implements a retry policy that follows one of the strategies I listed earlier. If you are a .NET developer like myself, you may be familiar with the Polly library. Golang has a library called Retry, and there are numerous third-party libraries for Python and Java.

Another question that rises is, would it be better to use some HttpClient package to handle the requests, like Refit in combination with Polly. But then again, it seems Refit also uses the HttpClient factory, which was a bad thing according to the previous?

D'Oh! Sorry, not PolySharp. I meant Polly. Too many similarly-named libraries!

Everywhere I've worked with AGs, we've worked with the application team to add retry logic to help make things a bit more seamless to end users. There are libraries out there that can make this pretty easy - Polly is one that I've used a few times, but there are others.

Fallacy 1: The network is reliable. If system 2 works perfectly well, but is not accessible for service 1 due to network issues, service 2 is still unavailable. This is why timeouts, service breakers and retry policies exist. A great tool for .NET to handle common network issues is Polly, but even when using a tool like this, the network is still not completely reliable.

I see. Then I don't have any better idea right now, but I do want to suggest that if your script is mostly API calls and you want to be able to deal with failures then take a look at the polly library: https://github.com/App-vNext/Polly

Runs in containers nicely, has good integration with Kafka, RabbitMQ, gRPC, etc. for Microservices communication. Implements resiliency patterns you'd want in Microservices via Polly. Has a decent Dependency Injection framework built in by default.

It's in no way hidden. But I use Polly all the time.

Depending if the sender or the reciever is down, you can also try Polly http://www.thepollyproject.org/