large-dataset Open-Source Projects

disk.frame

5 593 0.0 R

Fast Disk-Based Parallelized Data Manipulation Framework for Larger-than-RAM Data
ChartFx

25 477 9.3 Java

A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.

Project mention: Any way to monitor JavaFX thread | /r/JavaFX | 2023-12-07

For micro-benchmark-type measurements I added a few utility classes to chart-fx that can plot measurements from within the application in real time, e.g., live benchmarks.

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

large-dataset related posts

Do you code from memory? Or do you reference things?

1 project | /r/rstats | 31 Mar 2022
How can I read in only two columns from a massive 10+ GB tab file?

1 project | /r/rstats | 20 Jan 2022
Data cleaning/ analysis 100-200 million rows of data. Is this doable in R, or is there another program I should try instead?

2 projects | /r/rstats | 12 Oct 2021
is it possible to have my enviroment objects and work with them on my local drive instead of RAM?

1 project | /r/Rlanguage | 3 Jul 2021
We Test PCIe 4.0 Storage: The AnandTech 2021 SSD Benchmark Suite

1 project | news.ycombinator.com | 2 Feb 2021
A note from our sponsor - InfluxDB
www.influxdata.com | 17 May 2024

Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →