How to turn my data processing code into a pipeline?

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/dataengineering

Our great sponsors
  • Scout APM - Less time debugging, more time building
  • SonarQube - Static code analysis for 29 languages.
  • SaaSHub - Software Alternatives and Reviews
  • ploomber

    The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

    If you're working with .py or .ipynb I recommend ploomber, there's lots of flexibility on the input/output formats and you can interactively check out your data and work with Git. There's also an automatic tool that converts it to a pipeline for you via the H2 headings.

  • soorgeon

    Convert monolithic Jupyter notebooks 📙 into maintainable Ploomber pipelines. 📊

    If you're working with .py or .ipynb I recommend ploomber, there's lots of flexibility on the input/output formats and you can interactively check out your data and work with Git. There's also an automatic tool that converts it to a pipeline for you via the H2 headings.

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts