stats VS PDL-Graphics-Gnuplot

> do not believe in the possibility of a "General Theory of Productivity." I'm highly skeptical of attempts to quantify the precise relationship between error discovery stage and cost in a way that is generalizable, although I think it might be possible given a large group of engineers using a highly homogenous process, tools, and accounting. Google is pretty close to this (common dev infrastructure across tens of thousands of engineers), and even across Google this kind of generalization would be extremely difficult.

I don't think you are incorrect, but I think a lot of the aspirants behind ESE just want to have a better sense of what works and what doesn't; I'd even welcome negative results! The current state of things is to read 100 opinionated people and their blog posts. And given enough time, you'll encounter someone who swears that after drinking their morning coffee and jumping on one foot for 1 min, they enter a VRChat standup with their team and hit max flow. There's just so little knowledge right now about what works and what doesn't that I'd welcome more clarity, especially negative results.

> As a result, academic research into productivity can be difficult to generalize

I think defects are what we should measure for, not productivity because of the subjectivity of measuring productivity. But even measuring defects is complicated. The best way I see to measure defects is to ask a Team Under Test to document bugs that they encounter along with resolution times, but this is not only expensive, but something I doubt most corporations will be willing to share outside of their walls. Perhaps open source projects can try to store this data, like curl's stats [1].

[1]: https://github.com/curl/stats

Here is a link to the PDL book <http://pdl.perl.org/content/pdl-book-toc.html>.

I can share some examples of using PDL:

- Demos of basic usage <https://metacpan.org/release/ETJ/PDL-2.050/source/Demos/Gene...>

- Image analysis <https://nbviewer.ipython.org/github/zmughal/zmughal-iperl-no...> (I am also the author of IPerl, so if you have questions about it, let me know. My top priority with IPerl right now is to make it easy to install.)

- Physics calculations <https://github.com/wlmb/Photonic>

- Access to GSL functions for integration and statistics (with comparisons to SciPy and R): <https://gist.github.com/zmughal/fd79961a166d653a7316aef2f010...>. Note how PDL can take an array of values as input (which gets promoted into a PDL of type double) and then returns a PDL of type double of the same size. The values of that original array are processed entirely in C once they get converted to a PDL.

- Example of using Gnuplot <https://github.com/PDLPorters/PDL-Graphics-Gnuplot/blob/mast...>.

---

Just to give a summary of how PDL works relative to XS:

PDL allows for creating numeric ndarrays of any number of dimension of a specific type (e.g., byte, float, double, complex double) that can be operated on by generalized functions. These functions are compiled using a DSL called PP that generates multiple XS functions by taking a signature that defines the number of dimensions that the function operates over for each input/output variable and adding loops around it. These loops are quite flexible and can be made to work in-place so that no temporary arrays are created (also allows for doing pre-allocation). The loops will run multiple times over that same piece of memory --- this is still fast unless you have many small computations.

And if you do have many small computations, the PP DSL is available for the user to use as well so if they need to take a specific PDL computation written in Perl, they can translate the innermost loop into C and then it can do the whole computation in one loop (a faster data access pattern). There is a book for that as well called "Practical Magick with C, PDL, and PDL::PP -- a guide to compiled add-ons for PDL" <https://arxiv.org/abs/1702.07753>.

---

I'm also active on the `#pdl` IRC channel on <https://www.irc.perl.org/>, so feel free to drop by.