kaggle-environments VS Airflow

If you are interested in Data Science, you may want to look at Kaggle competitions. https://www.kaggle.com/competitions

Secondly, if you feel like you do not have enough skills or a lack of practice answering problem statements, there are a lot of good websites where you can find interesting projects. I would recommend starting participating in some Kaggle competitions or download some free Google datasets and start playing with them.

For reference: Kaggle is a Google product. You can see the list of current competitions here.

You could try Kaggle competitions, or participating in forecasting markets (as you stated) is another option. You don't need any specific skill set to be a forecaster, the rules of the bet are stipulated and from there it's just based on your ability to predict the outcome. You could also try your hand at investing in the stock market, or try and make money betting on sports games. If you're very good at this stuff I'm sure you can make a lot of money doing it. The thing to keep in mind is that generally video games are much much easier than real life

As to the specifics of your projects, that's up to you. Try browsing Kaggle; check out some of the work we have on The Pudding; check out some journalism examples to see what you can try to build on or improve.

You might want to look at kaggle competitions.

This approach means the API doesn't change, i.e., you can just replace the S3CopyObjectOperator instances with S3CopyOperator instances. Additionally, we only perform the extra work of doing the multipart upload when the simpler method is insufficient. The trade-off is that we're inefficient if almost every object is larger than 5GB because we're doing a "useless" API call first. As usual, it depends. A similar approach has been discussed in this Github Issue in the Airflow repository.

helm repo add apache-airflow https://airflow.apache.org

We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic needs to be separated. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages.

According to their README: https://github.com/apache/airflow#approach-to-dependencies-o...

> Nope, me too. The whole Repo network thing is not User facing at all.

There are some user-facing parts: You can find the fork network and some related bits under repo insights. (The UX is not great.)

https://github.com/apache/airflow/forks?include=active&page=...

There are several tools available that can help manage these workflows. Apache Airflow is a platform designed to programmatically author, schedule, and monitor workflows.

From what I've seen, there are sort of two paths. I'll provide a well known example from each.

1. lang specific distributed task library

For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

* https://github.com/celery/celery

Or lower level:

* https://github.com/dask/dask

2. DAG Workflow systems

There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

* https://github.com/apache/airflow

Job scheduling is an important part of data management as it enables regular data updates and cleanups. In a data platform, it is often undertaken by workflow orchestration tools like Apache Airflow and Apache Dolphinscheduler. However, adding another component to the data architecture also means investing extra resources for management and maintenance. That's why Apache Doris 2.1.0 introduces a built-in Job Scheduler. It is strategically more tailored to Apache Doris, and brings higher scheduling flexibility and architectural simplicity.

Instead of the custom orchestrator I used, a proper orchestration tool should replace it like Apache Airflow, Dagster, ..., etc.

An integral part of an ML project is data acquisition and data transformation into the required format. This involves creating ETL (extract, transform, load) pipelines and running them periodically. Airflow is an open source platform that helps engineers create and manage complex data pipelines. Furthermore, the support for Python programming language makes it easy for ML teams to adopt Airflow.

Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities.