Using a disk-backed Redis alternative to reduce AWS S3 bill

• cachenator

  1 92 1.0 Go

  Distributed, sharded in-memory cache and proxy for S3

  

• SQLAlchemy

  123 8,807 9.7 Python

  The Database Toolkit for Python

  

Hadn't heard of MinIO. We looked into another S3 caching solution that someone linked in comments here, but it came down to we already used SSDB in production. Also, a personal preference of mine is to keep all logic in Python. For ex: We use SQLAlchemy as the source of truth for our relational database schema, and we use Alembic to manage database schema changes in Python. We even keep the sharding of Postgres tables in Python [1]. It fits with my preference to keep this caching logic also in Python, by reading/writing directly to the cache and S3 from the Python app and workers.

[1] https://github.com/sqlalchemy/sqlalchemy/issues/6055#issueco...

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• goseaweedfs

  9 113 0.0 Go

  A complete Golang client for SeaweedFS

(I work on SeaweedFS) How about using SeaweedFS? https://github.com/chrislusf/seaweedfs

With your dedicated server, the latency is consistent, No API/network cost. Extra data can be tiered to S3.

Basically it is a key-file store.

https://github.com/chrislusf/seaweedfs/wiki/Filer-as-a-Key-L...

https://github.com/chrislusf/seaweedfs/wiki/Cloud-Tier

Suggest a related project