Kedro

A Python framework for creating reproducible, maintainable and modular data science code. (by kedro-org)

Kedro Alternatives

Similar projects and alternatives to Kedro

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better Kedro alternative or higher similarity.

Kedro reviews and mentions

Posts with mentions or reviews of Kedro. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-11-05.
  • What are examples of well-organized data science project that I can see on Github?
    6 projects | reddit.com/r/datascience | 5 Nov 2022
  • Dabbling with Dagster vs. Airflow
    7 projects | news.ycombinator.com | 14 Sep 2022
    An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.
  • What are some good DS/ML repos where I can learn about structuring a DS/ML project?
    3 projects | reddit.com/r/datascience | 27 Feb 2022
    For the lazy ones out there, here's the link to their github repo.
  • Kedro – Creating reproducible, maintainable and modular data science code
    4 projects | news.ycombinator.com | 22 Jan 2022
  • [Discussion] Applied machine learning implementation debate. Is OOP approach towards data preprocessing in python an overkill?
    3 projects | reddit.com/r/MachineLearning | 3 Nov 2021
    I'd focus more on understanding the issues in depth, before jumping to a solution. Otherwise, you would be adding hassle with some - bluntly speaking - opinionated and inflexible boilerplate code which not many people will like using. You mention some issues: non-obvious to understand code and hard to execute and replicate. Bad code which is not following engineering best practices (ideas from SOLID etc.) does not get better if you force the author to introduce certain classes. You can suggest some basics (e.g. common code formatter, meaningful variables names, short functions, no hard-coded values, ...), but I'm afraid you cannot educate non-engineers in a single day workshop. I would not focus on that at first. However, there is no excuse for writing bad code and then expecting others to fix. As you say, data engineering is part of data science skills, you are "junior" if you cannot write reproducible code. Being hard to execute and replicate is theoretically easy to fix. Force everyone to (at least hypothetically) submit their code into a testing environment where it will be automatically executed on a fresh machine. This will mean that at first they have to exactly specify all libraries that need to be installed. Second, they need to externalize all configuration - in particular data input and data output paths. Not a single value should be hard-coded in code! And finally they need a *single* command which can be run to execute the whole(!) pipeline. If they fail on any of these parts... they should try again. Work that does not pass this test is considered unfinished by the author. Basically you are introducing an automated, infallible test. Regarding your code, I'd really not try that direction. In particular even these few lines already look unclear and over-engineered. The csv format is already hard-coded into the code. If it changes to parquet you'd have to touch the code. The processing object has data paths fixed for which is no reason in a job which should take care of pure processing. Export data is also not something that a processing job should handle. And what if you have multiple input and output data? You would not have all these issues if you had kept to most simple solution to have a function `process(data1, data2, ...) -> result_data` where dataframes are passed in and out. It would also mean to have zero additional libraries or boilerplate. I highly doubt that a function `main_pipe(...)` will fix the malpractices some people may do. There are two small feature which are useful beyond a plain function though: automatically generating a visual DAG from the code and quick checking if input requirements are satisfied before heavy code is run. You can still put any mature DAG library on top, which probably already includes experience from a lot of developers. Not need to rewrite that. I'm not sure which one is best (metaflow, luigi, airflow, ... https://github.com/pditommaso/awesome-pipeline no idea), but many come with a lot of features. If you want a bit more scaffolding to easier understand foreign projects, you could look at https://github.com/quantumblacklabs/kedro but maybe that's already too much. Fix the "single command replication-from-scratch requirement" first.
    3 projects | reddit.com/r/MachineLearning | 3 Nov 2021
    Maybe check out kedro? It is a framework for doing what you want and a lot more.
  • Data Engineering in The Terminal | How I Kedro
    4 projects | dev.to | 18 Aug 2021
    kedro
  • Github Discussion: What is your favorite Data Science Repo?
    4 projects | reddit.com/r/datascience | 24 Jul 2021
    If you like cookiecutter-data-science, have a look at kedro https://github.com/quantumblacklabs/kedro
  • What is the best structured ds project you have seen?
    4 projects | reddit.com/r/datascience | 16 Apr 2021
    Another one of my personal faves is Kedro. Great ETL framework made especially for data scientists.
  • Ask HN: Who is hiring? (April 2021)
    21 projects | news.ycombinator.com | 1 Apr 2021
    Kedro, Quantumblack Labs | Python Software Engineer | London | REMOTE currently, ONSITE ? | Full-time

    Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering best-practice and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning.

    Kedro is listed on the 2020 ThoughtWorks Technology Radar (https://www.thoughtworks.com/radar/languages-and-frameworks/...) and the 2020 Data & AI Landscape (https://mattturck.com/data2020/) and has been framed as the "React for Data Science".

    Check out our repository: https://github.com/quantumblacklabs/kedro

    Please apply here mentioning Kedro + HN in your application: https://rb.gy/1njxa1

    We are at an exciting juncture in the maturing ML and MLOps space. We are keen to expand our function set through more sophisticated deployments and integrations with other ecosystem tools.

  • A note from our sponsor - Sonar
    www.sonarsource.com | 1 Dec 2022
    Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work. Learn more →

Stats

Basic Kedro repo stats
21
7,837
9.5
6 days ago
Write Clean Python Code. Always.
Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
www.sonarsource.com