safetensors VS llama.cpp

The ML field is doing work in that area: https://github.com/huggingface/safetensors

FYI the file format, safetensors, was proposed, developed and maintained by HF, and involved people from groups such as Eleuther and Stability for external security audits.

https://github.com/huggingface/safetensors https://huggingface.co/blog/safetensors-security-audit

Thank you for note on this. I had not heard there were already trojan horse malware being slipped into tensor files as python scripts. Apparently torch pickle uses eval on the tensor file with no filter.

Heard surprisingly little commentary on this topic. The full explanation of how Safetensors are "Safe" can be found from the developer at: https://github.com/huggingface/safetensors/discussions/111

Here the developer goes into some detail about what kinds of protections .safetensor files have : https://github.com/huggingface/safetensors/discussions/111

Use the safetensors format, which allows safe persistence and loading of models for common libraries - TensorFlow, PyTorch, JAX, etc. We went through external audits in the last few months (blog post). The current direction will be to have this as the default format.

I've been diving in lately and while it's not efficient, the only way to do manage is to create a new conda/mamba environment, or a custom Docker image for all the conflicting packages.

For safety and speed, you should prefer the safetensor format: https://huggingface.co/docs/safetensors/speed

If you know what you are doing you can do your own conversions: https://github.com/huggingface/safetensors or for safety, https://huggingface.co/spaces/diffusers/convert

GitHub - huggingface/safetensors: Simple, safe way to store and distribute tensors

if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp

the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...

And you can search for any GGUF on huggingface

Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182

For anyone interested in exploring this, llama.cpp has an example implementation here:

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

Getting started with LLMs can be intimidating. In this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like llama.cpp and KitOps.

well https://github.com/ggerganov/llama.cpp/issues/6849