polars VS loadtxt

This is because 0.1 is in actuality the floating point value value 0.1000000000000000055511151231257827021181583404541015625, and thus 1 divided by it is ever so slightly smaller than 10. Nevertheless, fpround(1 / fpround(1 / 10)) = 10 exactly.

I found out about this recently because in Polars I defined a // b for floats to be (a / b).floor(), which does return 10 for this computation. Since Python's correctly-rounded division is rather expensive, I chose to stick to this (more context: https://github.com/pola-rs/polars/issues/14596#issuecomment-...).

https://github.com/pola-rs/polars/releases/tag/py-0.19.0

That turned out to be related to pola-rs/polars#11912, and this linked comment provided a deceptively simple solution - use PARSE_DECLTYPES when creating the connection:

It really depends on what part of Numpy you're using. You can easily leave Numpy's text parsing in the dust. And if you're doing element-wise operations on arrays, you can easily see 2-3x improvement with just numba.

Amdahl's Law will catch up with you really fast as you add threads with this strategy, but it's simple and is amenable to formats where you may have a delimiter in the middle of a record. For situations where you need maximum scaling and don't have the possibility of delimiters scattered into records, you can use the strategy I used to implement a faster numpy.loadtxt: https://github.com/saethlin/loadtxt/blob/master/src/inner.rs#L84 The general idea is that you divide the file among thread boundaries by splitting it on byte boundaries, then seeking from that byte offset to the end of the next record. This gets you non-interleaved sections so there's no duplicate parsing.