Pandas VS NumPy

Pandas is a go-to tool for tabular data management, processing, and analysis in Python, but sometimes you may want to go from pandas to SQL.

panda’s

> The big thing pandas has going for it is that it's already been through this field testing. All the bugs have been ironed out by the hundreds of thousands of users.

At this very moment pandas github repo has 1563 open issues labeled with a bug tag [0]. So much for "all the bugs have been ironed out".

[0] https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3...

Python is the main programming language I use nowadays. In particular numpy and pandas are of course extremely useful. I also use biopython package - a collection of software tools for biological computation written in Python by an international group of researchers and developers.

Below is the python Task that transforms the tweets list into a Pandas dataframe, then dumps it in our MinIO Object Storage as a CSV file:

It was rewarding to dig into SQLite a bit while solving this puzzle, so I figured this would be a good opportunity to learn a bit more about pandas too! So how would I adapt this working SQL solution to pandas?

All the data has been extracted from the official Tor Metrics website, and using Python with the Pandas library I've cleaned the data. Finally, the visualizations have been made with Tableau.

If you did that you could use a built-in library like csv to read and parse the file or you could use a 3rd party library like Pandas. Alternatively, you could store your file as json:

NumPy: Scientific computing library and I know this one is the most popular especially in Data Science.

Python is the main programming language I use nowadays. In particular numpy and pandas are of course extremely useful. I also use biopython package - a collection of software tools for biological computation written in Python by an international group of researchers and developers.

If you want to start a project from scratch, I prefer to start with a very basic virtual environment and add the packages I need as I go along. My basic framework usually consists of: Python NumPy Pandas MatplotLib & sometimes Seaborn

Since you mention 2D arrays, did you consider using NumPy? It has tons of useful functions for array operations.

> If you look at something like numpy functions, so many of them share the exact same parameter definitions. What if you could write def log(standard-exp-params) instead of having to write them out every single time?

They're not actually written out every time, the issue is mostly documentary (and it would be nice if Python or Sphinx ever had a good solution). And numpy actually has a bunch of generators for that e.g. https://github.com/numpy/numpy/blob/45bc13e6d922690eea43b9d8... handles filling in the common bits of documentation for the ufuncs.

Python se pouziva vsude a nema specializaci. Asi jedina oblast kde vylozene vynika je Data Science a AI.

Not any more. They rewrote the Fortran parts: https://github.com/numpy/numpy

Definitely. However, the equivalent for the array multiplication in numpy looks like this:

https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c... + https://github.com/numpy/numpy/blob/22e683d84f2584a6f9a57b2c...

To know what they do, you need the source for INPLACE_GIVE_UP_IF_NEEDED (https://github.com/numpy/numpy/blob/b222eb66c79b8eccba39f46f...) and PyArray_GenericInplaceBinaryFunction (I don't even know where that's coming from, it's not defined in Numpy, maybe it's part of the Python interface?).

In the end, both are unreadable in their own way. I personally prefer the Rust version above to the macrofied C version that's in Numpy but that's a matter of taste. I'd also trust the safe Rust implementations more than the C implementations because of the memory management guarantees Rust provides, though I suppose for simple operations like multiplication it'll be easier to make the program safe enough in C.

The frames' vector data is generated using a sketch made with my vsketch framework (it involves loading GeoJSON data, processing it with Shapely, and projecting it in 3D with NumPy).

worth a look https://github.com/numpy/numpy/issues/14412