Whats something hot rn or whats going to be next thing we should focus on in data engineering?

• materialize

  117 5,567 10.0 Rust

  The data warehouse for operational workloads. (by MaterializeInc)

  

In terms of cutting edge tech, I'd say the companies in the data streaming space are doing some pretty cool stuff e.g https://materialize.com/

• monosi

  20 320 0.0 Python

  Open source data observability platform

  

Ah ok cool, well I guess you can say a lot of these tools that are becoming big with the modern data stack provide some form of automation. E.g Fivetran / Airbyte extract data on an automated schedule, then you have dbt with the transformations, and then the reverse ETLs like Hightouch / Census that run on an automated a schedule as well. I think it's pretty much becoming somewhat of a standard now to have, e.g. with what I'm building we included a scheduler for automation from the start.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• ploomber

  121 3,374 7.4 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

Ploomber :) We launched on HN yesterday!

• projects

  19 77 4.7 Jupyter Notebook

  Sample projects using Ploomber. (by ploomber)

  

Yes! (tell your friend). You can write shell scripts so you can execute that 2002 code :) You can test it locally and then run it in AWS Batch/Argo. Here's an example

Suggest a related project