How do you guys handle pandas and its sh*tty data type inference

• csv_log_cleaner

  2 2 6.3 Rust

  Clean CSV files to conform to a type schema by streaming them through small memory buffers using multiple threads and logging data loss.

Sounds like it could be more of a data cleansing problem you're facing than a data inference one. Even a single non-numerical value in a million rows of numbers will necessarily mess up type inference for the whole column. I work with a lot of CSVs and that's one of the issues we have to spend a huge amount of time dealing with. I even ended up writing this open source tool to handle the cleansing: https://github.com/ambidextrous/csv_log_cleaner

• dtype_diet

  1 34 3.8 Jupyter Notebook

  Tries to shrink your Pandas column dtypes with no data loss so you have more spare RAM

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• CSVLint

  44 134 7.6 C#

  CSV Lint plug-in for Notepad++ for syntax highlighting, csv validation, automatic column and datatype detecting, fixed width datasets, change datetime format, decimal separator, sort data, count unique values, convert to xml, json, sql etc. A plugin for data cleaning and working with messy data files.

There's also the CSV Lint plug-in for Notepad++ which can detect datatypes, and then you can do CSV Lint > Generate metadata > Python script. Although idk it might not work correctly for all datetime datatypes.

Suggest a related project