LLaMA-Factory VS qsv

Depends what model you want to train, and how well you want your computer to keep working while you're doing it.

If you're interested in large language models there's a table of vram requirements for fine-tuning at [1] which says you could do the most basic type of fine-tuning on a 7B parameter model with 8GB VRAM.

You'll find that training takes quite a long time, and as a lot of the GPU power is going on training, your computer's responsiveness will suffer - even basic things like scrolling in your web browser or changing tabs uses the GPU, after all.

Spend a bit more and you'll probably have a better time.

[1] https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#...

Thanks for the detailed feedback @snidane!

As maintainer of qsv, here's my reply:

- Given qsv's rapid release cycle (173 releases over three years), the auto-update check is essential at the moment. Once we reach 1.0, I'll turn it off. For now, given your feedback, I've only made it check 10% of the time.

- Pivot is in the backlog and I'll be sure to add unpivot when I implement it. (https://github.com/jqnatividad/qsv/issues/799)

- I'll add a dedicated summing command with the group by (-by) and window by (-over) capability (https://github.com/jqnatividad/qsv/issues/1514). Do note that `stats` has basic sum as @ezequiel-garzon pointed out.

- With the `enum` command, qsv can achieve what you proposed with `laminate`. E.g. qsv enum --new-column newcol --constant newconstant mydata.csv --output laminated-data.csv

- With the cat rowskey command, qsv can already concatenate files with mismatched headers.

- other file formats. qsv supports parquet, csv, tsv, excel, ods, datapackage, sqlite and more (see https://github.com/jqnatividad/qsv/tree/master#file-formats). Fixed-format though is not supported yet and quite interesting, and have added it to the backlog (https://github.com/jqnatividad/qsv/issues/1515)

- as to "enable embedding outputs of commands", qsv is composable by design, so you can use standard stdin/stdout redirection/piping techniques to have it work with other CLI tools like jq, awk, etc.

Finally, just released v0.120.0 that already incorporates the less aggressive self-update check. https://github.com/jqnatividad/qsv/releases/tag/0.120.0

qsv [1] also has a sqlp command which lets you run Polars SQL queries (even on multiple files). Here I'll send the csv data from stdin (represented by -) and then (optionally) pipe the output to the table command for formatting. The shape of the result is also printed to stderr (the (4, 2) below).

[1] https://github.com/jqnatividad/qsv

$ echo 'Name,Department,Salary

It works quite well IMHO. Using the mlua crate, I’ve managed to integrate Luau as a very powerful data-wrangling DSL for qsv (https://github.com/jqnatividad/qsv)

Maybe this might be better done using this? https://github.com/jqnatividad/qsv

https://github.com/jqnatividad/qsv is another option.