Working with more than 10gb csv

1. sqlitestudio

   1 47 5,768 9.9 C

   A free, open source, multi-platform SQLite database manager.

   

   https://sqlitestudio.pl is awesome, super easy to set up and pull in CSVs

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. modin

   2 11 10,057 8.5 Python

   Modin: Scale your Pandas workflows by changing a single line of code

   

   Modin should fit. It implements Pandas APIs with e.g. Ray as backend. https://github.com/modin-project/modin

4. spyql

   3 23 922 0.0 Jupyter Notebook

   Query data on the command line with SQL-like SELECTs powered by Python expressions

   

   You can import the data into a PostgreSQL/MySQL/SQLite/... database and then query the database. However, even with the right choice of indexes, it might take a while to run queries on a table with hundreds of millions of records. You can easily import your data to these databases with SpyQL: $ spyql "SELECT * FROM csv TO sql(table=my_table_name) | sqlite3 my.db" (you would need to create the table my_table_name before running the command).

Suggest a related project