data-infrastructure

Open-source projects categorized as data-infrastructure
Edit details
Language: + Python + Go + Rust + Java + TypeScript
Topics: Postgresql deet Python Kubernetes Database
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
Sponsored

Top 17 data-infrastructure Open-Source Projects

  • postgres-operator

    Postgres operator creates and manages PostgreSQL clusters running in Kubernetes

    • Project mention: Run PostgreSQL. The Kubernetes Way | news.ycombinator.com | 2023-09-22

    yes, precisely. It's UI part that's broken, which cannot list snapshots. Issue is here, no fix since 2020, sadly: https://github.com/zalando/postgres-operator/issues/937

  • postgres-operator

    Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service. (by CrunchyData)

    • Project mention: No disk space crashloop but pod healthy · Issue #3788 · CrunchyData/postgres-operator | /r/Health2020 | 2023-12-09

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • kubeblocks

    KubeBlocks is an open-source control plane that runs and manages databases, message queues and other data infrastructure on K8s.

    • Project mention: Open source db platform for vector db engines, qdrant, milvus, weaviate all in 1 | news.ycombinator.com | 2024-01-02

  • tensorbase

    TensorBase is a new big data warehousing with modern efforts.

  • spilo

    Highly available elephant herd: HA PostgreSQL cluster using Docker

    • Project mention: Why PostgreSQL High Availability Matters and How to Achieve It | news.ycombinator.com | 2023-06-14

    one of the solutions which made it pretty simple for us to run postgresql in a ha environment (mostly in k8s, but works standalone as well) is zalandos patroni: https://github.com/zalando/patroni it's really solid and worked for us for a few years already.

    or for k8s their operator: https://github.com/zalando/postgres-operator (docker image: https://github.com/zalando/spilo) we've also tried other operators which were easier to get started, but they failed miserably (crunchyrolls operator is basically based on the zalando one)

  • Nakadi

    A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

  • stream-unzip

    Python function to stream unzip all the files in a ZIP archive on the fly

    • Project mention: Show HN: stream-unzip – now with an async interface | news.ycombinator.com | 2024-03-23

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • pipebird

    Pipebird is open source infrastructure for securely sharing data with customers.

  • mbtiles-s3-server

    Python server to on-the-fly extract and serve vector tiles from an mbtiles file on S3

  • stream-zip

    Python function to construct a ZIP archive on the fly

    • Project mention: Show HN: stream-zip – now with async support | news.ycombinator.com | 2024-02-24

  • stream-sqlite

    Python function to extract rows from a SQLite file while iterating over its bytes

  • panda_patrol

    • Project mention: Show HN: Data monitoring and profiling with 1 function call | news.ycombinator.com | 2023-12-13

  • streampq

    Python PostgreSQL adapter to stream results of multi-statement queries without a server-side cursor

  • iterable-subprocess

    Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed

    • Project mention: Shshsh is a bridge connects Python and shell | news.ycombinator.com | 2023-09-23

    My own attempt at bridging the Python and, well, maybe not quite shell but more subprocess, boundary: https://github.com/uktrade/iterable-subprocess.

  • stream-write-ods

    Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

    • Project mention: What If OpenDocument Used SQLite? | news.ycombinator.com | 2023-09-18

  • mirror-git-to-s3

    Python functions and CLI to mirror git repositories to S3

  • stream-read-ods

    Python function to extract data from an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

    • Project mention: What If OpenDocument Used SQLite? | news.ycombinator.com | 2023-09-18

    Shameless plug of a couple of Python libraries I’ve been involved with that work around memory issues of ODS files (for very specific use cases):

    https://github.com/uktrade/stream-read-ods

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

data-infrastructure related posts

Index

What are some of the best open-source data-infrastructure projects? This list will help you:

Project Stars
1 postgres-operator 3,961
2 postgres-operator 3,719
3 kubeblocks 1,633
4 tensorbase 1,423
5 spilo 1,307
6 Nakadi 948
7 stream-unzip 250
8 pipebird 167
9 mbtiles-s3-server 135
10 stream-zip 85
11 stream-sqlite 23
12 panda_patrol 21
13 streampq 8
14 iterable-subprocess 7
15 stream-write-ods 3
16 mirror-git-to-s3 2
17 stream-read-ods 1

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com