pachyderm
ShellCheck
Our great sponsors
pachyderm | ShellCheck | |
---|---|---|
8 | 488 | |
6,074 | 34,934 | |
0.3% | - | |
9.8 | 8.6 | |
1 day ago | 11 days ago | |
Go | Haskell | |
Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pachyderm
-
Open Source Advent Fun Wraps Up!
20. Pachyderm | Github | tutorial
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.
-
Show HN: We scaled Git to support 1 TB repos
There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
- pachyderm: Data-Centric Pipelines and Data Versioning
-
Awesome list of VCs investing in commercial open-source startups
Pachyderm - License prevents competition.
-
Airflow's Problem
I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.
[0] https://github.com/pachyderm/pachyderm
-
One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
Here is a demo run of lgtm on pachyderm
- Dud: a tool for versioning data alongside source code, written in Go
ShellCheck
-
Ask HN: Is there a GUI for bash shell?
ncurse, dialog, zenity[2]. i/o buffering may be an issue [3a,3b]
Assuming using same account, use history command to show past commands[0a, 0b]
'load random example' on shellcheck using own custom examples from history command.[1]
--------
[3a] : http://www.gnu.org/software/coreutils/manual/html_node/stdbu...
[3b] : http://unix.stackexchange.com/questions/25372/how-to-turn-of...
[2] : http//funprojects.blog/2021/01/25/zenity-command-line-dialogs/
[1] : http://www.shellcheck.net/
[0a] : http://www.tecmint.com/history-command-examples/
[0b] : http://www.tecmint.com/remember-linux-commands/
web based documentation: https://www.tecmint.com/linux-commands-cheat-sheet/
commands grouped by typical usage patterns : https://www.tecmint.com/essential-linux-commands/
-
DevSecOps with AWS- IaC at scale - Building your own platform - Part 1
... #************************** Terraform ************************************* ARG TERRAFORM_VERSION=1.7.3 RUN set -ex \ && curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin/ RUN set -ex \ && mkdir -p $HOME/.terraform.d/plugin-cache && echo 'plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"' > ~/.terraformrc #************************* Terragrunt ************************************* ARG TERRAGRUNT_VERSION=0.55.1 RUN set -ex \ && wget https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 -q \ && mv terragrunt_linux_amd64 /usr/local/bin/terragrunt \ && chmod +x /usr/local/bin/terragrunt #*********************** Terramate **************************************** ARG TERRAMATE_VERSION=0.4.5 RUN set -ex \ && wget https://github.com/mineiros-io/terramate/releases/download/v${TERRAMATE_VERSION}/terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && tar -xzf terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && mv terramate /usr/local/bin/terramate \ && chmod +x /usr/local/bin/terramate #*********************** tfsec ******************************************** ARG TFSEC_VERSION=1.28.5 RUN set -ex \ && wget https://github.com/aquasecurity/tfsec/releases/download/v${TFSEC_VERSION}/tfsec-linux-amd64 \ && mv tfsec-linux-amd64 /usr/local/bin/tfsec \ && chmod +x /usr/local/bin/tfsec \ && terragrunt --version #**********************Terraform docs ************************************ ARG TERRRAFORM_DOCS_VERSION=0.17.0 RUN set -ex \ && curl -sSLo ./terraform-docs.tar.gz https://terraform-docs.io/dl/v${TERRRAFORM_DOCS_VERSION}/terraform-docs-v${TERRRAFORM_DOCS_VERSION}-$(uname)-amd64.tar.gz \ && tar -xzf terraform-docs.tar.gz \ && chmod +x terraform-docs \ && mv terraform-docs /usr/local/bin/terraform-docs #********************* ShellCheck ***************************************** ARG SHELLCHECK_VERSION="stable" RUN set -ex \ && wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION?}/shellcheck-${SHELLCHECK_VERSION?}.linux.x86_64.tar.xz" | tar -xJv \ && cp "shellcheck-${SHELLCHECK_VERSION}/shellcheck" /usr/bin/ \ && shellcheck --version ...
-
Ask HN: Popular open source tool originally written in Haskell?
ShellCheck: https://github.com/koalaman/shellcheck
-
Google ZX – A tool for writing better scripts
If I want to write better shell scripts I usually run shellcheck and adjust accordingly or if I need facilities not provided by the shell i switch to a full fledged programming language. Ans oh yes, `sh` is present almost on every BSD and Linux box for free so I consider it an important thing to at least be comfortable with.
shellcheck: https://www.shellcheck.net/
-
How I use Nix in my Elm projects
When I run nix-shell at the root of the project it puts me in a Nix shell that contains, among other programs, caddy and shellcheck. Notice that in the shellHook I add the project's shell scripts to the PATH. So once I'm in the Nix shell I can, among other things:
- Ask HN: A Bash guide for Posix programmers?
-
Regex support to list modules in .cabal?
I have also seen some projects on github like ShellCheck which first make a library, expose all the modules and then simple add that do build-depends of the final executable. Is this the recommended approach than having just one executable and adding all the modules to other-modules:?
-
Shellcheck finds bugs in your shell scripts
The error checks can be pretty arcane:
https://github.com/koalaman/shellcheck/wiki/Checks
-
Is there a syntax checker?
Similar to for instance shellcheck to check the syntax of shell scripts, is there an equivalent for the set of roff commands typically used in a (Linux) man page? I'm aware that e.g. pandoc permits the conversion of an other format (e.g., org) to both roff man and roff ms.
- Shellcheck – finds bugs in your shell scripts
What are some alternatives?
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
bash-language-server - A language server for Bash
trivy - Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
shfmt - Dockernized shfmt. This formats shell script.
dud - A lightweight CLI tool for versioning data alongside source code and building data pipelines.
shellharden - The corrective bash syntax highlighter
beneath - Beneath is a serverless real-time data platform ⚡️
shfmt - A shell formatter (sh/bash/mksh)
typhoon-orchestrator - Create elegant data pipelines and deploy to AWS Lambda or Airflow
PowerShell - PowerShell for every system!
tsuru - Open source and extensible Platform as a Service (PaaS).
efm-langserver - General purpose Language Server