minio-py
seaweedfs
minio-py | seaweedfs | |
---|---|---|
9 | 34 | |
767 | 21,076 | |
2.5% | 1.0% | |
8.3 | 9.9 | |
about 1 month ago | 4 days ago | |
Python | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
minio-py
-
MinIO - Mock S3 in local development
Python
-
Upload data directly from the web into Minio Bucket
Alternatively you could look into presigned urls which can also be done in python.
-
Reading JSON files from stream into memory
The program is running a SQL query against a remote file. This example in MinIO docs demonstrates reading from the response stream. So far, so good.
- Help with making a request to the API?
-
How do you build a bare minimum feature?
>So far your instinct as product designer/manager/engineer might be to take what customers say they want at face value.
That would be the worst way to go about building anything. Taking solutions from customers as opposed to problems to identify jobs-to-be-done and reasons for non consumption isn't something someone who knows how to build products does.
The question then becomes how does someone who generally knows how to build the right things build the bare minimum feature, the original question and title of the post, given the constraints on time and resources cited in the first paragraphs.
One can do several things. For example, writing the description and code examples for a yet to be library and shop it around, see if it makes sense, then write the library.
That's what I did for example with this library: https://pypi.org/project/bmc after shopping it around here: https://github.com/minio/minio-py/issues/829#issuecomment-65...
I also do that for internal tools, libraries, SDKs. I'll send client code around and see if it makes sense to engineers. I also had non engineers who never coded use libraries to do something useful. I just give them a laptop and docs and see how they use the thing. If they can do it, I know programmers can.
To get back to the original problem, prioritizing can lead towards minimal features. We have a section called "Instances" in our issue templates. If a feature issue does not have several instances where the problem manifested for several people, we just won't do it. We need concrete examples of a problem being frequent and expensive / high impact (loss of work, or prevents work in the first place).
For example, I'll look at the analytics for our internal platform and see that my colleague who we built the thing for is not using it. Why, we ask. It turns out the Docker images are too large and take a lot of time. He loses patience. They contain several libraries that he does need.
We built a minimal image that can get him to start after 30 seconds instead of 2 minutes, and we saw usage increase.
Yesterday, I saw he was giving a demo using his local environement, not the product. I'll ask why. What's wrong. What sucks. Why did he use his laptop instead of the platform.
The general sentiment of the article is sound. Always observe, ask questions, look for the underlying problems and how frequently they happen, how expensive they are (as in what are they doing to solve that, is there someone working on it, does it cost opportunities, etc)
-
Ask HN: What point you keep while designing a website for MVP?
I used to write libraries and have non-technical people who never programmed follow the documentation and use them. My reference point is that if they could do it, then programmers will have no trouble doing it.
The next best thing is writing the doc before the actual code, and have others look at it and see if it makes sense, then I'd write the code that makes that possible.[0]
- [0]: https://github.com/minio/minio-py/issues/829#issuecomment-65...
-
Tell HN: Curb Your Necroposting Aversion
Yes. A case of it working well: https://github.com/minio/minio-py/issues/829
I found this issue for a problem I had with MinIO. I wrote a prototype: docs first and screencast on how the API would be. I then asked the other user if they had implemented it and if the API I was thinking about would suit them as it suited me. MinIO's CTO replied that it was the correct usage, but that it was not on the roadmap, which is understandable.
I then started implementing the actual commands I needed first (host, ls, etc). Some time later, someone tweeted at us and asked how they could use the library. I then put it on PyPI (Python Package Index). I checked upon that person recently and they told be they've been using it for months without a problem. The other person also had commented and said they've been using it.
https://pypistats.org/packages/bmc
It's a really, really, small library for a niche need, also asked people for feedback here: https://news.ycombinator.com/item?id=26540342
Now I'm getting feature requests on that issue, but I'll soon do it properly in a repo to make it easier for people.
-
Ask HN: How do I improve boring README page?
Thank you for these points. I agree. It's one of those tools that, if you have that problem, you know exactly why you need it.
This solves this issue: https://github.com/minio/minio-py/issues/829. MinIO is an S3 compatible object storage software. https://min.io/
Thanks again.
-
The Trouble with Cassandra: Why It's a Poor Choice for Object Store Metadata DB
We use it in anger at Splitgraph [0] to let people upload/download datasets and Postgres table snapshots (instead of storing them directly in S3).
Pros:
* Less platform-dependent. By self-managing it, we can also run deploys to GCP / Azure / Scaleway / other providers without writing a separate adapter for e.g. Azure Blob Storage.
* Python API [1] much more pleasant to use than boto3 (and can speak to normal S3). It doesn't do everything that boto3 does, but it supports everything we need (e.g. pre-signed URLs).
* minio server itself supports a large chunk of S3's functionality (e.g. SELECT API / AssumeRole / bucket versioning)
* Don't pay per request and for egress: this was a big deal since people might want to download large amounts of data from us (or make a bunch of small requests to download/upload a subset of data).
Cons:
* Have to manage own infrastructure. We run it on managed VMs so it's semi-managed, but we still have to provision block storage, set up backup policies etc.
* In a similar vein, scaling and availability all have to be DIY [2]. We haven't run into situations yet where Minio would be the bottleneck, but it might be something to keep in mind.
* Obviously not as seamless: you don't get things like Glacier or integration with other IAM.
[0] https://www.splitgraph.com/
[1] https://github.com/minio/minio-py
[2] https://docs.min.io/docs/distributed-minio-quickstart-guide....
seaweedfs
-
DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System
Whoops: WebDAV:
https://news.ycombinator.com/item?id=39417503
SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV
I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.
It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.
-
Billion File Filesystem
If you want/need to take out the metadata, there's some nice solutions for that https://github.com/seaweedfs/seaweedfs
-
SeaweedFS fast distributed storage system for blobs, objects, files and datalake
I posted this on https://github.com/seaweedfs/seaweedfs/discussions/5290
-
DuckDB + dbt for a serverless event correlation pipeline?
I like the idea of using SeaweedFS as an intermediate layer with object write notifications going to SQS, RabbitMQ, or a local file, which could also allow me to observe the changes to different files through a metric collection layer like Prometheus and Grafana.
-
Show HN: OpenSign – The open source alternative to DocuSign
> Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.
Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing
I've actually been looking for more open alternatives, but haven't found much.
Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)
There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/
The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs
- The Tailscale Universal Docker Mod
- SeaweedFS
- Google Cloud Storage FUSE
- Experience running rook-ceph in production/large clusters
-
First Homelab as a 19yr old Software Developer
SeaweedFS S3 Gateway for Joplin notes
What are some alternatives?
Seaweed File System - SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. [Moved to: https://github.com/seaweedfs/seaweedfs]
minio - The Object Store for AI Data Infrastructure
s3www - Serve static files from any S3 compatible object storage services (Let's Encrypt ready)
Ceph - Ceph is a distributed object, block, and file storage platform
msgraph-sdk-python-core - Microsoft Graph client library for Python
garage - (Mirror) S3-compatible object store for small self-hosted geo-distributed deployments. Main repo: https://git.deuxfleurs.fr/Deuxfleurs/garage
awesome-readme - A curated list of awesome READMEs
cubefs - cloud-native file store
minio-java - MinIO Client SDK for Java
GlusterFS - Web Content for gluster.org -- Deprecated as of September 2017
Apache AGE - Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL. [Moved to: https://github.com/apache/age]
MooseFS - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)