-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
As many have already mentioned this is more of data engineering than Data Science one. Try to build ETL pipelines for storing the data to a data lake or data warehouse(more organised). Make sure the pipelines are reliable and fall back mechanism to ensure consistency. Check out open source DBs https://clickhouse.tech (open-source OLAP database management system) or you can get started with Postgres as well. https://airbyte.io is an open source project which provides data integrations/pipelines.
As many have already mentioned this is more of data engineering than Data Science one. Try to build ETL pipelines for storing the data to a data lake or data warehouse(more organised). Make sure the pipelines are reliable and fall back mechanism to ensure consistency. Check out open source DBs https://clickhouse.tech (open-source OLAP database management system) or you can get started with Postgres as well. https://airbyte.io is an open source project which provides data integrations/pipelines.