SymPy VS ModelingToolkit.jl

Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover:

- From sympy: https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results...

- Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment.

There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results).

That's interesting. You should consider yourself lucky to have met Wolfram employees, as they are obviously vastly outnumbered by users of Mathematica.

I have not met any developers for either of these products but I know that SymPy has a huge list of contributors for a project of its size. See: https://github.com/sympy/sympy/blob/master/AUTHORS

You may not be hearing about SymPy users because SymPy is not a monolithic product. It is a library. If you know mathematicians big into using Python, they are probably aware of SymPy as it is the main attraction when it comes to symbolic computation in Python.

https://github.com/sympy/sympy/issues/9479 suggests that multivariate inequalities are still unsolved in SymPy, though it looks like https://github.com/sympy/sympy/pull/21687 was merged in August. This probably isn't yet implemented in C++ in SymForce yet?

bug report opened https://github.com/sympy/sympy/issues/25507

https://news.ycombinator.com/item?id=36463580

From https://news.ycombinator.com/item?id=36159017 :

> sympy.utilities.lambdify.lambdify() https://github.com/sympy/sympy/blob/a76b02fcd3a8b7f79b3a88df... :

>> """Convert a SymPy expression into a function that allows for fast numeric evaluation [with the CPython math module, mpmath, NumPy, SciPy, CuPy, JAX, TensorFlow, SymPy, numexpr,]*

From https://westurner.github.io/hnlog/#comment-19084622 :

> "latex2sympy parses LaTeX math expressions and converts it into the equivalent SymPy form" and is now merged into SymPy master and callable with sympy.parsing.latex.parse_latex(). It requires antlr-python-runtime to be installed. https://github.com/augustt198/latex2sympy https://github.com/sympy/sympy/pull/13706

ENH: 'generate a Jupyter notebook' (nbformat .ipynb JSON) function from this stem formula

https://en.wikipedia.org/wiki/Vectorization :

> Array programming, a style of computer programming where operations are applied to whole arrays instead of individual elements

> Automatic vectorization, a compiler optimization that transforms loops to vector operations

> Image tracing, the creation of vector from raster graphics

> Word embedding, mapping words to vectors, in natural language processing

> Vectorization (mathematics), a linear transformation which converts a matrix into a column vector

Vector (disambiguation) https://en.wikipedia.org/wiki/Vector

> Vector (mathematics and physics):

> Row and column vectors, single row or column matrices

> Vector space

> Vector field, a vector for each point

And then there are a number of CS usages of the word vector for 1D arrays.

Compute kernel: https://en.m.wikipedia.org/wiki/Compute_kernel

GPGPU > Vectorization, Stream Processing > Compute kernels: https://en.wikipedia.org/wiki/General-purpose_computing_on_g...

sympy.utilities.lambdify.lambdify() https://github.com/sympy/sympy/blob/a76b02fcd3a8b7f79b3a88df... :

> """Convert a SymPy expression into a function that allows for fast numeric evaluation [with the CPython math module, mpmath, NumPy, SciPy, CuPy, JAX, TensorFlow, SymPt, numexpr,]

pyorch lambdify PR, sympytorch: https://github.com/sympy/sympy/pull/20516#issuecomment-78428...

Sympytorch:

> Turn SymPy expressions into PyTorch Modules.

> SymPy floats (optionally) become trainable parameters. SymPy symbols are inputs to the Module.

sympy2jax https://github.com/MilesCranmer/sympy2jax :

> Turn SymPy expressions into parametrized, differentiable, vectorizable, JAX functions.

> All SymPy floats become trainable input parameters. SymPy symbols become columns of a passed matrix.

Look at sympy.isprime for a carefully-optimized pure-Python solution (though if gmpy2 is installed, which it usually is, it will use that instead after trying the easiest cases)

I'd use a modeling tool like https://mtk.sciml.ai/dev/ Using the standard library, you wouldn't need to come up with all equations yourself. Depending on the details of your use case, system identification as suggested before might be a faster approach though.

No, you do get type errors during runtime. The most common one is a MethodNotFound error, which corresponds to a dispatch not being found. This is the one that people then complain about for long stacktraces and as being hard to read (and that's a valid criticism). The reason for it is because if you do xy with a type combination that does not have a corresponding dispatch, i.e. (x::T1,y::T2) not defined anywhere, then it looks through the method table of the function, does not find one, and throws this MethodNotFound error. You will only get no error if a method is found. Now what can happen is that you can have a method to an abstract type, *(x::T1,y::AbstractArray), but `y` does not "actually" act like an AbstractArray in some way. If the way that it's "not an AbstractArray" is that it's missing some method overloads of the AbstractArray interface (https://docs.julialang.org/en/v1/manual/interfaces/#man-inte...), you will get a MethodNotFound error thrown on that interface function. Thus you will only not get an error if someone has declared `typeof(y) <: AbstractArray` and implemented the AbstractArray interface.

However, what Yuri pointed out is that there are some packages (specifically in the statistics area) which implemented functions like `f(A::AbstractArray)` but used `for i in 1:length(A)` to iterate through x's values. Notice that the AbstractArray interface has interface functions for "non-traditional indices", including `axes(A)` which is a function to call to get "the a tuple of AbstractUnitRange{<:Integer} of valid indices". Thus these codes are incorrect, because by the definition of the interface you should be doing `for i in axes(A)` if you want to support an AbstractArray because there is no guarantee that its indices go from `1:length(A)`. Note that this was added to the `AbstractArray` interface in the v1.0 change, which is notably after the codes he referenced were written, and thus it's more that they were not updated to handle this expanded interface when the v1.0 transition occurred.

This is important to understand because the criticisms and proposed "solutions" don't actually match the case... at all. This is not a case of Julia just letting anything through: someone had to purposefully define these functions for them to exist. And interfaces are not a solution here because there is an interface here, its rules were just not followed. I don't know of an interface system which would actually throw an error if someone does a loop `for i in 1:length(A)` in a code where `A` is then indexed by the element. That analysis is rather difficult at the compiler level because it's non-local: `length(A)` is valid since querying for the length is part of the AbstractArray interface (for good reasons), so then `1:length(A)` is valid since that's just range construction on integers, so the for loop construction itself is valid, and it's only invalid because of some other knowledge about how `A[i]` should work (this look structure could be correct if it's not used to `A[i]` but rather do something like `sum(i)` without indexing). If you want this to throw an error, the only real thing you could do is remove indexing from the AbstractArray interface and solely rely on iteration, which I'm not opposed to (given the relationship to GPUs of course), but etc. you can see the question to solving this is "what is the right interface?" not "are there even interfaces?" (of which the answer is, yes but the errors are thrown at runtime MethodNotFound instead of compile time MethodNotImplemented for undefined things, the latter would be cool for better debugging and stacktraces but isn't a solution).

This is why the real discussions are not about interfaces as a solution, they don't solve this issue, and even further languages with interfaces also have this issue. It's about tools for helping code style. You probably should just never do `for i in 1:length(A)`, probably you should always do `for i in eachindex(A)` or `for i in axes(A)` because those iteration styles work for `Array` but also work for any `AbstractArray` and thus it's just a safer way to code. That is why there are specific mentions to not do this in style guides (for example, https://github.com/SciML/SciMLStyle#generic-code-is-preferre...), and things like JuliaFormatter automatically flag it as a style break (which would cause CI failures in organizations like SciML which enforce SciML Style formatting as a CI run with Github Actions https://github.com/SciML/ModelingToolkit.jl/blob/v8.14.1/.gi...). There's a call to add linting support for this as well, flagging it any time someone writes this code. If everyone is told to not assume 1-based indexing, formatting CI fails if it is assumed, and the linter underlines every piece of code that does it as red, (along with many other measures, which includes extensive downstream testing, fuzzing against other array types, etc.) then we're at least pretty well guarded against it. And many Julia organizations, like SciML, have these practices in place to guard against it. Yuri's specific discussion is more that JuliaStats does not.

The thing to watch in the space of Simulink/Modelica is https://github.com/SciML/ModelingToolkit.jl . It's an acausal modeling system similar to Modelica (though extended to things like SDEs, PDEs, and nonlinear optimization), and has a standard library (https://github.com/SciML/ModelingToolkitStandardLibrary.jl) similar to the MSL. There's still a lot to do, but it's pretty functional at this point. The two other projects to watch are FunctionalModels.jl (https://github.com/tshort/FunctionalModels.jl, which is the renamed Sims.jl), which is built using ModelingToolkit.jl and puts a more functional interface on it. Then there's Modia.jl (https://github.com/ModiaSim/Modia.jl) which had a complete rewrite not too long ago, and in its new form it's fairly similar to ModelingToolkit.jl and the differences are more in the details. For causal modeling similar to Simulink, there's Causal.jl (https://github.com/zekeriyasari/Causal.jl) which is fairly feature-complete, though I think a lot of people these days are going towards acausal modeling instead so flipping Simulink -> acausal, and in that transition picking up Julia, is what I think is the most likely direction (and given MTK has gotten 40,000 downloads in the last year, I think there's good data backing it up).

And quick mention to bring it back to the main thread here, the DataDrivenDiffEq symbolic regression API gives back Symbolics.jl/ModelingToolkit.jl objects, meaning that the learned equations can be put directly into the simulation tools or composed with other physical models. We're really trying to marry this process modeling and engineering world with these "newer" AI tools.

In this case, yes, this should just be done numerically. But using symbolic transformations to optimize numeric code is also a really neat application of symbolic computing that doesn't get enough attention, imo. [This library](https://github.com/SciML/ModelingToolkit.jl), for example, uses symbolics to do sparsity detection, automatic derivative/gradient/jacobian/hessian calculations, index reduction, etc. to speed up numerical differential equation solving.

https://homes.cs.washington.edu/~thickstn/ctpg-project-page/...

That's all showing the raw iteration count to show that it algorithmically is faster, but the time per iteration is also fast for many reasons showcased in the SciMLBenchmarks routinely outperforming C and Fortran solvers (https://github.com/SciML/SciMLBenchmarks.jl). So it's excelling pretty well, and things like the automated discovery of black hole dynamics are all done using the universal differential equation framework enabled by the SciML tools (see https://arxiv.org/abs/2102.12695 for that application).

What we are missing however is that, right now these simulations are all writing raw differential equations so we do need a better set of modeling tools. That said, MuJoCo and DiffTaichi are not great physical modeling environments for building real systems, instead we would point to Simulink and Modelica as what are really useful for building real-world systems. So it would be cool if there was a modeling language in Julia which extends that universe and directly does optimal code generation for the Julia solvers... and that's what ModelingToolkit.jl is (https://github.com/SciML/ModelingToolkit.jl). That project is still pretty new, but there's already enough to show some large-scale models outperforming Dymola on examples that require symbolic tearing and index reduction, which is far more than what physical simulation environments used for non-scientific purposes (MuJoCo and DiffTaichi) are able to do. See the workshop for details (https://www.youtube.com/watch?v=HEVOgSLBzWA). And that's just the top level details, there's a whole Julia Computing product called JuliaSim (https://juliacomputing.com/products/juliasim/) which is then being built on these pieces to do things like automatically generate ML-accelerated components and add model building GUIs.

That said, MuJoCo and DiffTaichi have much better visualizations and animations than MTK. Our focus so far has been on the core routines, making them fast, scalable, stable, and extensive. You'll need to wait for the near future (or build something with Makie) if you want the pretty pictures of the robot to happen automatically. That said, Julia's Makie visualization system has already been shown to be sufficiently powerful for this kind of application (https://nextjournal.com/sdanisch/taking-your-robot-for-a-wal...), so we're excited to see where that will go in the future.

PDEs are difficult because you don't have a simple numerical definition over all PDEs because they can be defined by arbitrarily many functions. u' = Laplace u + f? Define f. u' = g(u) * Laplace u + f? Define f and g. Etc. To cover the space of PDEs you have to go symbolic at some point, and make the discretization methods dependent on the symbolic form. This is precisely what the ModelingToolkit.jl ecosystem is doing. One instantiation of a discretizer on this symbolic form is NeuralPDE.jl which takes a symbolic PDESystem and generates an OptimizationProblem for a neural network which represents the solution via a Physics-Informed Neural Network (PINN).

There's a very clear momentum for Julia here in this domain of modeling and simulation. With JuliaSim funding an entire modeling and simulation department within Julia Computing dedicated to building out an ecosystem that accelerates this domain and the centralization around the SciML tooling, this is an area where we absolutely have both a manpower and momentum advantage. We're getting many universities (PhD students and professors) involved on the open source side, while building out different commercial tools and GUIs on top of the open numerical core. The modeling and simulation domain itself is soon going to have its own SciMLCon since our developer community has gotten too large to just be a few JuliaCon talks: it needs its own days to fit everyone! Not only that, in many aspects we're not just moving faster but have already passed. Not in every way, there's still some important discussion in controls that needs to happen, but that's what the momentum is for.

Julia has great support for modeling, have a look at ModelingToolkit.jl. From the README: