pachyderm VS ESLint

20. Pachyderm | Github | tutorial

Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.

There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.

If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)

The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.

Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)

Pachyderm - License prevents competition.

I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.

[0] https://github.com/pachyderm/pachyderm

Here is a demo run of lgtm on pachyderm

Let's be honest - setting up tools for a new project can be a frustrating process. Especially when you want to jump straight to coding part. This is often the case with ESLint and Prettier, two popular tools in the JavaScript ecosystem that can sometimes interfere with each other when it comes to code formatting. Fortunately, there's a simple solution to this process, and it's called eslint-plugin-prettier.

Do you have the need to find and fix problems in your JavaScript? Then ESLint is for you. With over 120 million downloads each month, it's definitely one of the most popular projects out there. The latest version introduces a new configuration system, designed to simplify available configuration options. Old and deprecated rules have been removed, and a new configuration inspector has been added. Check out the blog for all the highlights, and a migration guide.

ESLint: A pluggable and configurable linter tool for identifying and reporting on patterns in JavaScript.

Automating code checks with static code analysis allows us to enforce code styling effectively. By integrating tools into our workflow, we can identify errors at an early stage, while coding instead of blocking us at the end. For instance, flake8 checks Python code for style and errors, eslint performs similar checks for JavaScript, and prettier automatically formats code to maintain consistency.

If you're a developer, you're surely familiar with Prettier and ESLint. With over 8 years of existence, they have established themselves as references in the JavaScript ecosystem.

eslint is used to avoid code errors

No lint errors: The committed code does not contain any lint errors (eslint).

Let’s walk through the steps for a one-time setup to configure husky pre-commit and pre-push hooks, ESLint with code styles conventions, prettier code formatter, and lint-staged. Husky automatically runs a script on each commit or push. This is useful for linting files to enforce code styles that keeps the entire code base following conventions.

GitHub | Website

ESLint: A pluggable and configurable linter tool for identifying and reporting on patterns in JavaScript.