s3-benchmark
warp
| s3-benchmark | warp | |
|---|---|---|
| 4 | 4 | |
| 776 | 493 | |
| - | 4.1% | |
| 0.0 | 6.8 | |
| 4 months ago | 15 days ago | |
| Go | Go | |
| MIT License | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s3-benchmark
- S3 Benchmark: Measure Amazon S3's performance from any location
- S3 Benchmark
-
Ask HN: Have you ever switched cloud?
There's another benchmark somewhere showing S3 can max out a 100Gbps instance.
https://github.com/dvassallo/s3-benchmark
Another potential issue is ListBucket rate limiting. If you have lots of small objects, you'll spend most of the time waiting to discover the names than transferring data
-
A distributed Posix file system built on top of Redis and S3
TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation).
S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB).
I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here.
[1] https://github.com/dvassallo/s3-benchmark
warp
-
MinIO HA and distributed setup on VMs
Once you have the setup, you can benchmark with https://github.com/minio/nperf and https://github.com/minio/dperf and if you really want to stress test you can use https://github.com/minio/warp. If those numbers are acceptable for your use case, you can move forward assuming you have some ability to predict / prepare for traffic patterns to ramp up in the future.
-
Minio Search and read file performance question
MinIO provides several built in tools for this via mc support diag {disk,object,net}. These tests are server specific. To test more thoroughly you can use https://github.com/minio/warp, but you will need sufficient client machines to push the traffic.
-
Kasten-io snapshot data export slow
If you have enough client machines to test, you can look at https://github.com/minio/warp but it is more for heavier testing for larger prod deployments. There is a nice utilty within the mc client itself, mc support perf will show you throughput and iops of the cluster.
-
K8s storage provider benchmarks round 2, part 4 - setting up the benchmarking tools
Agreed, can't beat not leaving your machine :). I'll make a note to get some Object gateway tests in -- what did you use for your s3 benchmarking? Did you just script it or use a tool like warp? I'd like to get as close to replicating your set up so I can compare numbers.
What are some alternatives?
s5cmd - Parallel S3 and local filesystem execution tool.
wgcf - 🚤 Cross-platform, unofficial CLI for Cloudflare Warp
juicefs - JuiceFS is a distributed POSIX file system built on top of Redis and S3.
k8s-storage-provider-benchmarks
awesome-go-storage - A curated list of awesome Go storage projects and libraries
hperf - Distributed HTTP Speed Test.
redisraft - A Redis Module that make it possible to create a consistent Raft cluster from multiple Redis instances.
stork - Stork - Storage Orchestration Runtime for Kubernetes
rpCheckup - rpCheckup is an AWS resource policy security checkup tool that identifies public, external account access, intra-org account access, and private resources.
containers-roadmap - This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).