Our great sponsors
-
seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I built https://github.com/anacrolix/squirrel for just this purpose. However you might find that batch inserting empty files might only be 10x faster or so than creating empty files directly on the filesystem if we are to believe the 10k/s the OP achieved.
If you want/need to take out the metadata, there's some nice solutions for that https://github.com/seaweedfs/seaweedfs
Related posts
- Distributed cache with client-side consistent hashing
- Seeking feedback on my first big Go project - minicache: a distributed cache with client-side consistent hashing, arbitrary cluster sizes, support for both HTTP/gRPC interfaces, secured with mTLS
- Distributed cache with client-side consistent hashing
- Embedded write-heavy on-disk cache, write-amplification
- Micro – A Go service development platform