timescaledb-insert-benchmarks
open-data
| timescaledb-insert-benchmarks | open-data | |
|---|---|---|
| 1 | 1 | |
| 14 | 74 | |
| - | - | |
| 8.8 | 6.4 | |
| about 2 months ago | 4 months ago | |
| Python | ||
| MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
timescaledb-insert-benchmarks
-
Loading a trillion rows of weather data into TimescaleDB
The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. Still quite big to upload.
The data is freely available from the [Climate Change Service](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...) which has a nice API but download speeds can be a bit slow.
[NCAR's Research Data Archive](https://rda.ucar.edu/datasets/ds633-0/) provides some of the data (as pre-generated NetCDF files) but at higher download speeds.
It's not super well documented but I hosted the Python scripts I used to download the data on the accompanying GitHub repository: https://github.com/ali-ramadhan/timescaledb-insert-benchmark...
open-data
-
Loading a trillion rows of weather data into TimescaleDB
Creator of Open-Meteo here. There is small tutorial to setup ERA5 locally: https://github.com/open-meteo/open-data/tree/main/tutorial_d...
Under the hood Open-Meteo is using a custom file format with time-series chunking and specialised compression for low-frequency weather data. General purpose time-series databases do not even get close to this setup.