digger
s5cmd
Our great sponsors
digger | s5cmd | |
---|---|---|
85 | 11 | |
2,670 | 2,324 | |
3.6% | 3.7% | |
9.9 | 7.3 | |
2 days ago | about 2 months ago | |
Go | Go | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
digger
-
Ask HN: Should we build support for more CI platforms, or features for Actions?
Currently, Github Actions is de-facto the only fully supported CI platform in Digger, we’ve been building it as a CI-agnostic tool (https://github.com/diggerhq/digger) from get go. We keep getting requests to support more CI systems on our community slack and over Github issues (https://github.com/diggerhq/digger/issues/81).
Unlike other automation tools for Terraform, Digger doesn’t run jobs on the server; instead it uses your CI (like Actions) as a compute backend. This is more secure and also much cheaper if you use your own runners in your CI.
But each CI and each VCS is ever so slightly different; and we are now at a crossroads - Should we build support for more CI platforms, or more features for GitHub Actions? We’d love any thoughts/inputs!
-
Ask HN: Should open-source projects allow disabling telemetry?
We just had a user submit an issue and a PR to revert the changes we made earlier that remove the option to disable telemetry. We feel like it’s a fair ask to share usage data with authors of an open-source tool that’s early in the making; but the user’s viewpoint is also perfectly understandable. Are we in the wrong here?
https://github.com/diggerhq/digger/issues/1179
Surely we aren’t the first open-source company to face this dilemma. We don’t want to alienate the community; but losing visibility of usage doesn’t sound great either. Give people the “more privacy” button and most are going to press it. Is there a happy medium?
-
Terraform drift detection and remediation - a primer
Detecting and managing drift in Terraform is a multifaceted process. The use of Terraform commands such as terraform refresh and terraform plan plays a critical role in identifying drifts. Additionally, periodic monitoring of the infrastructure using these commands can aid in early detection and prevention of larger issues. Tools like Digger and Terraform Cloud offer dedicated drift detection and remediation mechanisms. These tools provide continuous monitoring and notifications, enabling teams to stay informed about the state of their infrastructure.
-
Typical challenges faced while setting up CI/CD for Terraform at scale
Star us on GitHub | Check out Docs | Blog | Slack
-
GitHub issues from top Open Source Golang Repositories that you should contribute to
I would be extremely grateful if you could give us a star & share your thoughts in the comments section below https://github.com/diggerhq/digger
-
Tools used by the top 1% of Platform Engineers and their Commercial Open Source Alternatives
Check Digger's repo on GitHub
-
5 Open Source tools written in Golang that you should know about
Digger is an Open Source Infrastructure as Code management tool that helps orchestrate IaC such as Terraform & OpenTofu within GitHub Actions. Digger reuses compute used for application code so that you don't overpay for 3rd party managed compute for IaC. This approach eliminates the duplication of CI/CD infrastructure such as compute, jobs, and logs, and reduces security concerns by keeping sensitive data within the CI job. Digger's integration with existing CI systems offers scalability by leveraging on-demand compute resources and enhances security by confining data within the existing CI environment.
-
Top 10 terraform tools you should know about.
Digger is an Open Source IaC management platform that allows you to orchestrate terraform/OpenTofu in your CI/CD system. It helps you resue async jobs infrastructure with compute, orchestration, logs, etc of your existing CI. Digger also has a pro version built on top of Digger’s community edition. Digger’s “bring your own compute” ensures that users have private runners by defualt and don’t have to pay for it additionally. Digger pro gives team leads, managers and IaC practitioners dashboards, Drift Detection, RBAC via OPA policies and concurrency so they can help guide the team.
-
Restricting apply permissions
We have this issue filed by a user and cannot quite decide internally what to do about it. It is clear that allowing everyone who comments to trigger apply is problematic. And we have this covered by the RBAC via OPA feature that allows users to set granular rego policies for who can do what. But it can also be viewed as an overkill to introduce the whole policy-as-code setup for a basic restriction like this.
- Digger - Open Source Terraform & OpenTofu automation tool for teams (Runs within GitHub Actions so you don't pay for 3rd party compute)
s5cmd
-
GitHub issues from top Open Source Golang Repositories that you should contribute to
s5cmd - Extended character support for s3 compatible backend
-
Migrate 5 TB S3 bucket from one AWS account to another
I've used a tool in the past called s5cmd to copy millions of objects, and it was strikingly fast: https://github.com/peak/s5cmd
-
Those using AWS, have you ever tried to use AWS Transfer Family to transfer files into an S3 bucket? Can I use python to make these uploads, and if so how do I set it up in aws?
Some folks say https://github.com/peak/s5cmd is faster than the two options above.
- Gcloud storage: up to 94% faster data transfers for Cloud Storage
- Faster way to empty S3 buckets?
-
A Dockerfile for Perl 5.36 / Alpine, with working SSL
RUN mkdir /tmp/output && cd /tmp/output RUN wget --no-check-certificate https://github.com/peak/s5cmd/releases/download/v1.2.1/s5cmd_1.2.1_Linux-64bit.tar.gz RUN tar xvzf s5cmd_1.2.1_Linux-64bit.tar.gz && mv s5cmd /usr/bin/s5cmd && rm -rf /tmp/output && rm s5cmd_1.2.1_Linux-64bit.tar.gz
-
DataSync Vs AWS S3 sync?
Not that I’ve seen but you might checkout https://github.com/peak/s5cmd
-
S3/100gbps question
I like to use https://github.com/peak/s5cmd
-
Downloading files from S3 with multithreading and Boto3
Excellent walkthrough, love boto. We’ve recently been using s5cmd which we’ve found is ridiculously faster than boto without any extra boto tricks.
https://github.com/peak/s5cmd
- How to download millions of files from S3? (AWS CLI stops working after 1st million)
What are some alternatives?
terrakube - Open source IaC Automation and Collaboration Software.
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
atlantis - Terraform Pull Request Automation
s4cmd - Super S3 command line tool
terrateam - Terraform automation for teams. Purpose-built for GitHub.
s3-proxy - S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
s3-benchmark - Measure Amazon S3's performance from any location.
otf - An open source alternative to terraform enterprise.
kool - From local development to the cloud: web apps development with containers made easy.
iTorrent - Torrent client for iOS 9.3+
aptly - aptly - Debian repository management tool