mach-nix VS lm-evaluation-harness

Another way if the above doesn't work is to use mach-nix:https://github.com/DavHau/mach-nix

For python, I can highly recommend mach-nix. Makes it really easy to also keep a requirements.txt to stay compatible with non-nix-users.

The ESP32 toolchain is quite cumbersome to install - even under NixOS. The wiki tells you to create a shell.nix that creates a python virtual env at the first execution. I used DavHau's mach-nix to create esp-idf.nix and subsequently micropython-esp32.nix.

mach-nix and friends are pretty good, but I've found that the expectations of all the commonly-used data science stuff is pretty antithetical to the Nix Way. I'm not sure if it's still the case, but last I tried, Hydra wasn't building CUDA (since it's non-free), so I had to compile that and e.g. TensorFlow and PyTorch. Very painful, even on a beefy system.

I don't do much Python, but usually everything works just fine with mach-nix. I would try something like this: ``` { pkgs ? import (fetchTarball "https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz") { } }:

There are some sifferent/new tools for adding your own Python packages these days. It's still not truly solved, but one of these new package generation tools might serve your better:

poetry2nix: https://github.com/nix-community/poetry2nix

dream2nix: https://nix-community.github.io/dream2nix/guides/getting-sta...

mach-nix: https://github.com/DavHau/mach-nix

pip2nix: https://github.com/nix-community/pip2nix

pynixify: https://github.com/cript0nauta/pynixify

The tools available to you at the time (pypi2nix and maybe python2nix, if it was a long time ago) have been abandoned in favor of the newer tools, I think chiefly poetry2nix but I'm not sure.

There's still the Nixpkgs buildPythonPackage stuff, I think, if your goal is to upstream a lib into Nixpkgs. But if you just want to build your own Python applications and vendorize the deps (e.g., for work), you might try one of the tools above, which weren't available 3+ years ago. Maybe Nixy Python users and developers can reply with some of their experiences using those tools :)

Have a look at mach-nix which is a small utility library for nix to build Python packages declaratively.

final: prev: { xonsh = prev.xonsh.overrideAttrs (old: rec { version = "0.12.4"; src = final.fetchFromGitHub { owner = "xonsh"; repo = "xonsh"; rev = version; sha256 = "0kdps0gf0767zy0fs6qn39rv4z3x7ck0qz1pzx6962593171yk8b"; }; propagatedBuildInputs = prev.xonsh.propagatedBuildInputs ++ [final.python3Packages.virtualenv]; }); python39 = prev.python39.override { self = prev.python39; packageOverrides = python_final: python_prev: { prompt-toolkit = python_prev.prompt-toolkit.overrideAttrs (old: rec { version = "3.0.29"; src = final.python3Packages.fetchPypi { pname = "prompt_toolkit"; inherit version; sha256 = "sha256-vWQPYOjOzXTw3CSXE9QzrOLdxitl7gf5bTWOCxUrbqc="; }; }); }; }; # Using mach-nix to fetch unpackaged xontrib plugins # adapted from https://github.com/NixOS/nixpkgs/issues/75786#issuecomment-873654103 mach-nix = import (builtins.fetchGit { url = "https://github.com/DavHau/mach-nix/"; ref = "refs/tags/3.4.0"; }) { pkgs = final; }; xonsh_pyenv = final.mach-nix.mkPython { requirements = '' xontrib-fzf-widgets xonsh-direnv ''; }; xonsh_with_plugins = final.xonsh.overrideAttrs (old: { propagatedBuildInputs = old.propagatedBuildInputs ++ final.xonsh_pyenv.python.pkgs.selectPkgs final.xonsh_pyenv.python.pkgs; }); }

The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)

Fair question.

Evaluate refers to the phase after training to check if the training is good.

Usually the flow goes training -> evaluation -> deployment (what you called inference). This project is aimed for evaluation. Evaluation can be slow (might even be slower than training if you're finetuning on a small domain specific subset)!

So there are [quite](https://github.com/microsoft/promptbench) [a](https://github.com/confident-ai/deepeval) [few](https://github.com/openai/evals) [frameworks](https://github.com/EleutherAI/lm-evaluation-harness) working on evaluation, however, all of them are quite slow, because LLM are slow if you don't have infinite money. [This](https://github.com/open-compass/opencompass) one tries to speed up by parallelizing on multiple computers, but none of them takes advantage of the fact that many evaluation queries might be similar and all try to evaluate on all given queries. And that's where this project might come in handy.

I haven't run this yet, but I'm aware of Eleuther AI's evaluation harness EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models. (github.com) and GPT-4 -based evaluations like lm-sys/FastChat: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and FastChat-T5. (github.com)

Updates: Just finished final evaluation (additional metrics) on https://github.com/EleutherAI/lm-evaluation-harness and have averaged the results for orca-mini-v2-13b. The average results for the Open LLM Leaderboard are not that great, compare to initial metrics. The average is now 0.54675 which put this model below then many other 13b out there.

Hey u/The-Bloke Appreciate the quants! What is the degradation on the some benchmarks. Have you seen https://github.com/EleutherAI/lm-evaluation-harness. 3-bit and 2-bit quant will really be pushing it. I don't see a ton of evaluation results on the quants and nice to see a before and after.

I am primarily looking for results of running the MMLU evaluation on modern large language models. I have been able to find some data here https://github.com/EleutherAI/lm-evaluation-harness/tree/master/results and will be asking them if/when, they can provide any additional data.

I evaluated orca_mini_v2_7b on a wide range of tasks using Language Model Evaluation Harness from EleutherAI.

There is the Language Model Evaluation Harness project which evaluates LLMs on over 200 tasks. HuggingFace has a leaderboard tracking performance on a subset of these tasks.

https://github.com/EleutherAI/lm-evaluation-harness

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...