NumPy VS Pandas

As is the case with most Python libraries, it is open-source and free-to-use, making it easily accessible by anyone willing to learn machine learning, and it is built upon other open-source libraries within Python, like SciPy for advanced scientific operations, NumPy for efficient numerical computations, Matplotlib for data visualization, and Cython for increased efficiency and speed, similar to that of C/C++.

I hit a similar issue in 2017 which is still the case today: Python's builtin `random.shuffle` destroys numpy arrays passed into it [0]. This is apparently a design limitation within numpy and cannot be detected or fixed, so it still stands today.

[0] https://github.com/numpy/numpy/issues/10215

The AI Service will be built using aiohttp (asynchronous Python web server) and integrates PyTorch, Hugging Face Transformers, numpy, pandas, and scikit-learn for financial data analysis.

In my testing, I found that when checking out 500 commits sequentially from the numpy repository, disabling this feature required 13.8 seconds to complete on average across 10 runs. Enabling this feature took on average 11.2 seconds across 10 runs. Not an astounding difference in testing, but if core.fsmonitor can save me 2.6 seconds per 500 commits, on a project with 37,775 commits that could add up to a time savings of 211.54 seconds, or 3 minutes and 32 seconds! More testing on my end needs to be done if this feature scales linearly, but for now I will keep it on and use version 1 of the tool.

Readers of this blog who are familiar with LAPACK are likely to not be intimately familiar with the wild world of web technologies. For those coming from the world of numerical and scientific computation and have familiarity with the scientific Python ecosystem, the easiest way to think of stdlib is as an open source scientific computing library in the mold of NumPy and SciPy. It provides multi-dimensional array data structures and associated routines for mathematics, statistics, and linear algebra, but uses JavaScript, rather than Python, as its primary scripting language. As such, stdlib is laser-focused on the web ecosystem and its application development paradigms. This focus necessitates some interesting design and project architecture decisions, which make stdlib rather unique when compared to more traditional libraries designed for numerical computation.

numpy

This library provides functions for working in domain of linear algebra, fourier transform, matrices and arrays.

https://github.com/numpy/numpy/issues/26510#issuecomment-229...

And now that I look at that again I realize I forgot to finish that up!

Libraries for data science and deep learning that are always changing

# Read the content of nda.txt try: import os, types import pandas as pd from botocore.client import Config import ibm_boto3 def __iter__(self): return 0 # @hidden_cell # The following code accesses a file in your IBM Cloud Object Storage. It includes your credentials. # You might want to remove those credentials before you share the notebook. cos_client = ibm_boto3.client(service_name='s3', ibm_api_key_id='api-generated', ibm_auth_endpoint="https://iam.cloud.ibm.com/identity/token", config=Config(signature_version='oauth'), endpoint_url='https://s3.direct.us-south.cloud-object-storage.appdomain.cloud') bucket = 'your-bucket-referenced-here' object_key = 'nda__da__crxq8b2hmy.txt' # load data of type "text/plain" into a botocore.response.StreamingBody object. # Please read the documentation of ibm_boto3 and pandas to learn more about the possibilities to load the data. # ibm_boto3 documentation: https://ibm.github.io/ibm-cos-sdk-python/ # pandas documentation: http://pandas.pydata.org/ streaming_body_1 = cos_client.get_object(Bucket=bucket, Key=object_key)['Body'] with open("nda.txt", "r") as f: nda_content = f.read() print("Content of nda.txt has been read.") except FileNotFoundError: print("Error: nda.txt not found in the current directory.") nda_content = "" # Initialize knowledge source content_source = CrewDoclingSource( file_paths=["..."] )

As with any web scraping or data processing project, I had to write a fair amount of code to clean this up and shape it into a format I needed for further analysis. I used a combination of Pandas and regular expressions to clean it up (full code here).

Can this be merged into pandas?

Pandas does not currently install tqdm by default.

pandas-dev/pandas//pyproject.toml [project.optional-dependencies] https://github.com/pandas-dev/pandas/blob/main/pyproject.tom...

Dask solves for various adjacent problems; IDK if pandas, dask, or dask-cudf would be faster with async?

Dask docs > Scheduling > Dask Distributed (local) https://docs.dask.org/en/stable/scheduling.html#dask-distrib... :

> Asynchronous Futures API

Dask docs > Deploy Dask Clusters; local multiprocessing poll, k8s (docker desktop, podman-desktop,), public and private clouds, dask-jobqueue (SLURM,), dask-mpi:

They are when the only reason they are flagged as security updates is because some a single group deems a very rare, obscure edge case as a HIGH severity vuln when in practice it rarely is => this leads to having to upgrade a minor version of a library that ends up causing breaking changes.

This is the recent thread I'm down. Pandas 2.2 broke SQLalchemy backwards compatibility: https://stackoverflow.com/questions/38332787/pandas-to-sql-t... + https://github.com/pandas-dev/pandas/issues/57049#issuecomme...

Python’s Growth in Data Work and AI: Python continues to lead because of its easy-to-read style and the huge number of libraries available for tasks from data work to artificial intelligence. Tools like TensorFlow and PyTorch make it a must-have. Whether you’re experienced or just starting, Python’s clear style makes it a good choice for diving into machine learning. Actionable Tip: If you’re new to Python, try projects that combine data with everyday problems. For example, build a simple recommendation system using Pandas and scikit-learn.

This tutorial provides a concise and foundational guide to exploring a dataset, specifically the Sample SuperStore dataset. This dataset, which appears to originate from a fictional e-commerce or online marketplace company's annual sales data, serves as an excellent example for learning and how to work with real-world data. The dataset includes a variety of data types, which demonstrate the full range of capabilities that the Pandas library, written in Python can offer.

Data preprocessing and manipulation: Libraries like Pandas solve for the messy, real-world challenge of efficiently wrangling and cleaning large datasets. Without it, you'd be reinventing functionality for basic tasks like merging, filtering, or aggregating data.

The AI Service will be built using aiohttp (asynchronous Python web server) and integrates PyTorch, Hugging Face Transformers, numpy, pandas, and scikit-learn for financial data analysis.

Pandas