Optimization.jl VS SciPy

Interesting response. I develop the Julia SciML organization https://sciml.ai/ and we'd be more than happy to work with you to get wrappers for PRIMA into Optimization.jl's general interface (https://docs.sciml.ai/Optimization/stable/). Please get in touch and we can figure out how to set this all up. I personally would be curious to try this out and do some benchmarks against nlopt methods.

There are also Optimization and Nonconvex , which seem like umbrella packages and I am not sure what methods to use inside these packages. Any help on these?

> but would you say most packages follow or enforce SemVer?

The package ecosystem pretty much requires SemVer. If you just say `PackageX = "1"` inside of a Project.toml [compat], then it will assume SemVer, i.e. any version 1.x is non-breaking an thus allowed, but not version 2. Some (but very few) packages do `PackageX = ">=1"`, so you could say Julia doesn't force SemVar (because a package can say that it explicitly believes it's compatible with all future versions), but of course that's nonsense and there will always be some bad actors around. So then:

> Would enforcing a stricter dependency graph fix some of the foot guns of using packages or would that limit composability of packages too much?

That's not the issue. As above, the dependency graphs are very strict. The issue is always at the periphery (for any package ecosystem really). In Julia, one thing that can amplify it is the fact that Requires.jl, the hacky conditional dependency system that is very not recommended for many reasons, cannot specify version requirements on conditional dependencies. I find this to be the root cause of most issues in the "flow" of the package development ecosystem. Most packages are okay, but then oh, I don't want to depend on CUDA for this feature, so a little bit of Requires.jl here, and oh let me do a small hack for OffSetArrays. And now these little hacky features on the edge are both less tested and not well versioned.

Thankfully there's a better way to do it by using multi-package repositories with subpackages. For example, https://github.com/SciML/GalacticOptim.jl is a global interface for lots of different optimization libraries, and you can see all of the different subpackages here https://github.com/SciML/GalacticOptim.jl/tree/master/lib. This lets there be a GalacticOptim and then a GalacticBBO package, each with versioning, but with tests being different while allowing easy co-development of the parts. Very few packages in the Julia ecosystem actually use this (I only know of one other package in Julia making use of this) because the tooling only recently was able to support it, but this is how a lot of packages should be going.

The upside too is that Requires.jl optional dependency handling is by far and away the main source of loading time issues in Julia (because it blocks precompilation in many ways). So it's really killing two birds with one stone: decreasing package load times by about 99% (that's not even a joke, it's the huge majority of the time for most packages which are not StaticArrays.jl) while making version dependencies stricter. And now you know what I'm doing this week and what the next blog post will be on haha.

I guess it is a rite of passage to rewrite it. I'm doing it for SciPy too together with Propack in [1]. Somebody already mentioned your repo. Thank you for your efforts.

[1]: https://github.com/scipy/scipy/issues/18566

Fortran codes have caused many problems for the Python package Scipy, and some of them are now being rewritten in C: e.g., https://github.com/scipy/scipy/pull/19121. Not only does R have many Fortran codes, there are also many R packages using Fortran codes: https://github.com/r-devel/r-svn, https://github.com/cran?q=&type=&language=fortran&sort=. Modern Fortran is a fine language but most legacy Fortran codes use the F77 style. When I update the R package quantreg, which uses many Fortran codes, I get a lot of warning messages. Not sure how the Fortran codes in the R ecosystem will be dealt with in the future, but they recently caused an issue in R due to the lack of compiler support for Fortran: https://blog.r-project.org/2023/08/23/will-r-work-on-64-bit-arm-windows/index.html. Some renowned packages like glmnet already have their Fortran codes rewritten in C/C++: https://cran.r-project.org/web/packages/glmnet/news/news.html

There are several lessons here: a) vanilla conda-forge numpy and scipy versions come with openblas, and it works pretty well, b) do not use netlib unless your matrices are small and you need to do a lot of SVDs, or idek why c) Apple's veclib/accelerate is super fast, but it is also numerically unstable. So much so that the scipy's devs dropped any support of it back in 2018. Like dang. That said, they are apparently are bring it back in, since the 13.3 release of macOS Ventura saw some major improvements in accelerate performance.

First, if you read through that scipy issue (https://github.com/scipy/scipy/issues/18118 ) the author was willing and able to relicense PRIMA under a 3-clause BSD license which is perfectly acceptable for scipy.

For the numerical recipes reference, there is a mention that scipy uses a slightly improved version of Powell's algorithm that is originally due to Forman Acton and presumably published in his popular book on numerical analysis, and that also happens to be described & included in numerical recipes. That is, unless the code scipy uses is copied from numerical recipes, which I presume it isn't, NR having the same algorithm doesn't mean that every other independent implementation of that algorithm falls under NR copyright.

Reading the discussions under a previous thread titled "More Descent, Less Gradient"( https://news.ycombinator.com/item?id=23004026 ), I guess people might be interested in PRIMA ( www.libprima.net ), which provides the reference implementation for Powell's renowned gradient/derivative-free (zeroth-order) optimization methods, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA.

PRIMA solves general nonlinear optimizaton problems without using derivatives. It implements Powell's solvers in modern Fortran, compling with the Fortran 2008 standard. The implementation is faithful, in the sense of being mathmatically equivalent to Powell's Fortran 77 implementation, but with a better numerical performance. In contrast to the 7939 lines of Fortran 77 code with 244 GOTOs, the new implementation is structured and modularized.

There is a discussion to include the PRIMA solvers into SciPy ( https://github.com/scipy/scipy/issues/18118 ), replacing the buggy and unmaintained Fortran 77 version of COBYLA, and making the other four solvers available to all SciPy users.

if that makes your eyes bleed, what do you think about this? https://github.com/scipy/scipy/blob/main/scipy/special/specfun/specfun.f (heh)