prima VS SciPy

The developer of PRIMA here.

If you use method "cobyla" from scipy.optimize.minimize, then PRIMA already performs far better (in terms of the number of function evaluations). See the comparison at https://github.com/libprima/prima#improvements .

The bugs are indeed only a secondary reason: they can only be triggered under special situations. They may not affect your usage at all (when it does affect you, the consequence is catastrophophic).

1. nagfor Release 7.1(Hanzomon) Build 7149 released on March 5, 2024, fixed all the bugs spotted, but introduced an ICE when compiling PRIMA ( http://www.libprima.net ). The ICE has nothing to do with half-precision real, because it occurs when PRIMA is configured to use single or double precision. It can be reproduced by

```

git clone https://github.com/libprima/prima.git && cd prima && git checkout ec42cb0 && cd fortran/examples/lincoa && make ntest

```

2. nagfor 7.2 released on 6 March, 2024 included neither the ICE nor the fixes for the bugs.

My goal is to rewrite this library for derivative-free optimization: https://github.com/libprima/prima

A native port is indeed planned. However, since we are talking about a project of about 10K lines of code, such a port will not be delivered very soon.

In fact, native implementations of PRIMA in Python, MATLAB, C++, Julia, and R will all be done in the future. See https://github.com/libprima/prima#other-languages . But it takes time. PRIMA has been a one-man project since it started three yearss ago. Community help is greatly needed.

Thanks.

GitHub repo of the project: https://github.com/libprima/prima

It sounds like this was a difficult task. The motivation to fulfill Prof. Powell's request and help the community of derivative-free optimization users must have been strong. Congratulations on your achievement!

From the GitHub README:

> In the past years, while working on PRIMA, I have spotted a dozen of bugs in reputable Fortran compilers and two bugs in MATLAB. Each of them represents days of bitter debugging, which finally led to the conclusion that it was not a problem in my code but a flaw in the Fortran compilers or in MATLAB. From a very unusual angle, this reflects how intensive the coding has been.

> The bitterness behind this "fun" fact is exactly why I work on PRIMA: I hope that all the frustrations that I have experienced will not happen to any user of Powell's methods anymore. I hope I am the last one in the world to decode a maze of 244 GOTOs in 7939 lines of Fortran 77 code — I have been doing this for three years and I do not want anyone else to do it again.

https://github.com/libprima/prima#a-fun-fact

I guess it is a rite of passage to rewrite it. I'm doing it for SciPy too together with Propack in [1]. Somebody already mentioned your repo. Thank you for your efforts.

[1]: https://github.com/scipy/scipy/issues/18566

Fortran codes have caused many problems for the Python package Scipy, and some of them are now being rewritten in C: e.g., https://github.com/scipy/scipy/pull/19121. Not only does R have many Fortran codes, there are also many R packages using Fortran codes: https://github.com/r-devel/r-svn, https://github.com/cran?q=&type=&language=fortran&sort=. Modern Fortran is a fine language but most legacy Fortran codes use the F77 style. When I update the R package quantreg, which uses many Fortran codes, I get a lot of warning messages. Not sure how the Fortran codes in the R ecosystem will be dealt with in the future, but they recently caused an issue in R due to the lack of compiler support for Fortran: https://blog.r-project.org/2023/08/23/will-r-work-on-64-bit-arm-windows/index.html. Some renowned packages like glmnet already have their Fortran codes rewritten in C/C++: https://cran.r-project.org/web/packages/glmnet/news/news.html

There are several lessons here: a) vanilla conda-forge numpy and scipy versions come with openblas, and it works pretty well, b) do not use netlib unless your matrices are small and you need to do a lot of SVDs, or idek why c) Apple's veclib/accelerate is super fast, but it is also numerically unstable. So much so that the scipy's devs dropped any support of it back in 2018. Like dang. That said, they are apparently are bring it back in, since the 13.3 release of macOS Ventura saw some major improvements in accelerate performance.

First, if you read through that scipy issue (https://github.com/scipy/scipy/issues/18118 ) the author was willing and able to relicense PRIMA under a 3-clause BSD license which is perfectly acceptable for scipy.

For the numerical recipes reference, there is a mention that scipy uses a slightly improved version of Powell's algorithm that is originally due to Forman Acton and presumably published in his popular book on numerical analysis, and that also happens to be described & included in numerical recipes. That is, unless the code scipy uses is copied from numerical recipes, which I presume it isn't, NR having the same algorithm doesn't mean that every other independent implementation of that algorithm falls under NR copyright.

Reading the discussions under a previous thread titled "More Descent, Less Gradient"( https://news.ycombinator.com/item?id=23004026 ), I guess people might be interested in PRIMA ( www.libprima.net ), which provides the reference implementation for Powell's renowned gradient/derivative-free (zeroth-order) optimization methods, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA.

PRIMA solves general nonlinear optimizaton problems without using derivatives. It implements Powell's solvers in modern Fortran, compling with the Fortran 2008 standard. The implementation is faithful, in the sense of being mathmatically equivalent to Powell's Fortran 77 implementation, but with a better numerical performance. In contrast to the 7939 lines of Fortran 77 code with 244 GOTOs, the new implementation is structured and modularized.

There is a discussion to include the PRIMA solvers into SciPy ( https://github.com/scipy/scipy/issues/18118 ), replacing the buggy and unmaintained Fortran 77 version of COBYLA, and making the other four solvers available to all SciPy users.

if that makes your eyes bleed, what do you think about this? https://github.com/scipy/scipy/blob/main/scipy/special/specfun/specfun.f (heh)