Pandas VS Airflow

If you really want to store it all (labels included) in one data structure, you should look up pandas.

Another question you may be asking is: "What about skill players?" Well, what about them? Skill players are defined as players that consistently tote the rock. I was able to filter out skill player's performance in different combine events using pandas. For our purposes, the following positions (as listed on PRF) were considered 'skill players': WR, RB, QB, TE, DB, LB. In included linebackers but if you want to not include them, knock yourself out. It kind of only helps my case that the likes of Roquan Smith and Nakobe Dean don't count for the SEC. When only considering skill players, the SEC ranks 2nd to the Big 12 in 40-yard dash times. In the other combine events for which there is data, the SEC ranks first in none of them.

You glossed over the fact that `.pct_change` isn't actually "percent change" as documented. More fun reading: https://github.com/pandas-dev/pandas/issues/20752

I suggest pandas might be a great tool for you as you will be able to read write excel / csv files and process them and see how you get on.

Now to implement this one, we'll have to do some manipulation to our account values. Let's use the power of numpy to help us out here (oh and it's also the same in pandas too. We'll be using np.diff to take the returns of our account values and resampling them.

Increase code efficiency: Python is designed for ease of use and easy extension, but not performance. As a developer, the onus is on you to do more work so that the application executes less code. Whenever possible use vectorized library functions instead of loops. Python is successful in data science because of the pre-compiled code offered by data-appropriate libraries in the pydata stack such as pandas and numpy.

Built on top of Python, pandas is an open source data analysis and manipulation tool, similar to NumPy. While it relies on NumPy arrays for much of its manipulation and computation, pandas makes it easier to visualize and explore data, helping our team make better sense of the large amounts of data we work with on a daily basis.

Dataframes are not meant to be as flexible as lists in terms of extending the data they hold, dataframes are much more "deliberate". Ideally if you're trying to dynamically add data to a dataframe you should first collect all the data and then initialize the dataframe once. Or collect separate dataframes and concat them once. The pandas developers are even thinking about deprecating append (see here)

LOL, not sure if your joking or not but ... this sub is for a software package called Airflow (https://airflow.apache.org/), not physical airflow.

Figuring out how to migrate our existing Airflow deployment and DAGs to something more easily deployable to k8s

Anyone used Airflow (https://airflow.apache.org/) to automate all the things yet?

Just to offer one more, Apache Airflow is another program that offers job scheduling functionality: https://airflow.apache.org/

Moreover, configure and deploy the Luigi's Scheduler on a server / pod for production use is easy, while it might be not for other similar tools like Apache AirFlow.

Be automated with tools like dbt or Apache Airflow or integrated into a continuous integration (CI) process.

ListItem(name='Apache Airflow', website='https://airflow.apache.org/', category='Workflow Engine', short_description='Apache Airflow is an open-source workflow management platform for data engineering pipelines.'),

So, if you even want to use MLFlow to track your experiments, run the pipeline on Airflow, and then deploy a model to a Neptune Model Registry, ZenML will facilitate this MLOps Stack for you. This decision can be made jointly by the data scientists and engineers. As ZenML is a framework, custom pieces of the puzzle can also be added here to accommodate legacy infrastructure.

ZenML is an exercise in finding the right layer of abstraction for ML. Here, we treat pipelines as first-class citizens. This means that data scientists are exposed to pipelines directly in the framework, but not in the same manner as the data pipelines from the ETL space (Prefect, Airflow et al.). Pipelines are treated as experiments — meaning they can be compared and analyzed directly. Only when it is time to flip over to productionalization, can they be converted to classical data pipelines.

In this demo, we use the request itself as the trigger and begin computation immediately. But it may vary according to the nature of your application. Often, you might have to use a separate pipeline as well. In such scenarios, you may need technologies such as Apache Airflow or Prefect.