Ask HN: Best way to provide access to large data sets

• datasette

  187 8,955 9.3 Python

  An open source multi-tool for exploring and publishing data

I've been researching methods for exposing large data sets to users of my website, and came across the Atlassian Data Lake (https://www.atlassian.com/platform/analytics/what-is-atlassian-data-lake), and was curious - is this a thing that companies are doing now?

For personal projects and smaller scale non-dynamic data sets I've used Datasette (https://datasette.io/), but haven't seen anyone providing anything at scale beyond either an API for accessing data or some sort of periodic file export.

• roapi

  24 3,087 6.9 Rust

  Create full-fledged APIs for slowly moving datasets without writing a single line of code.

  

For smaller datasets then anywhere up to a few mb which isn't so bad reasonable with an API but in theory for historic data it could be up to several gb. I've not seen datasette go that high (IIRC it's a 1000 row return limit by default).

That's what got me intrigued with Atlassians offering, as data lakes tend to be something internal to a company, not something I've ever seen offered as an interaction point to users.

I've also tested out roapi [1] which is nice if the data is in some structured format already (Parquet/JSON)

[1] https://github.com/roapi/roapi

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project