ide VS ModelingToolkit.jl

docs (https://enso.org/docs/syntax). We've added many new libraries, so you can do many more things with it now. Oh, and we changed the name to Enso and got accepted to YC! :)

The problem we address is that data analysts still waste up to half of their time on repetitive manual work that can be automated [6]. To give one example, a company we're working with hires business users who use Excel to define data quality rules. These get manually translated to SQL, then manually translated to Python. This is not only error prone, it’s so slow that it takes them 90 days to introduce a single new rule. There’s 60 days’ worth of overhead in this process—it’s insane!

Years ago I (Wojciech) led the in-house development of visual effects (VFX) tools at a motion picture studio. We made tools like cloud renderers and smoke simulation engines. The artists using these tools did not have any programming background, yet they were designing complex algorithms for forces between particles, light subsurface scattering, things like that. Earlier generations of these tools had hundreds of config options, buttons, etc., for masses of different use cases, but this approach got way too complex and people eventually realized that it falls short when you need to do anything that the vendor did not think of. Nowadays they use node-based software (like the Houdini FX) which lets users draw algorithms as a sequence of data processing steps (these steps are often referred to as “nodes”). Later, when I was working in other industries and encountered the same rats’ nests of complex GUIs for solving data processing problems, I realized that the data analytics/science space was in need of the same breakthrough that we had already gone through in the VFX space.

Most visual programming languages / workflow-builders do not scale well because they don't let users express abstractions. Try to build a complex pipeline and you'll end with an unreadable spaghetti of connections—it's like coding a web app in the assembler. Enso is different because we allow you to build abstractions to manage the complexity. As a result, you never have more than 10-20 nodes on the stage in Enso (nodes are hierarchical). You can create custom data types, custom components (functions), catch errors, etc. All this works because under the hood, Enso is a real programming language. However, naive implementations of such systems are super slow. Each component may be built of hundreds, sometimes thousands of lower-level ones. The real trick is making these hierarchical components run fast. For that you need a dedicated compiler and a runtime system, and this is a hard technical space. Our system involves a dedicated JIT compiler based on GraalVM. For details, see https://enso.org/language#compiler. In case this is interesting for you, here is our podcast about how the compiler works under the hood: https://www.youtube.com/watch?v=BibjcUjdkO4.

Enso is interactive, meaning that we recompute the relevant parts of graphs as parameters change, which shortens feedback loops dramatically. Like a lot of people on HN, we were inspired by Bret Victor's classic talk on instant feedback: https://www.youtube.com/watch?v=8QiPFmIMxFc. We’ve also put a lot of effort into extensibility. You can add Java, JavaScript, R, and Python (soon also Ruby, Scala, Kotlin, Rust, and C) directly into Enso nodes without the need to write any wrappers and with a close-to-zero performance overhead.

Enso is open source. Our compiler code is at https://github.com/enso-org/enso and our GUI code at https://github.com/enso-org/ide. Our business model is based on selling domain specific libraries, on-premise installations with enhanced user permission management, and coming soon, a hosted solution called Enso Cloud, which will be our only non-open-source codebase. Since this is Hacker News, I should add that all our alpha releases collect anonymous usage statistics which we use to improve Enso and prepare it for a stable release. Full details about that are always in our release notes (https://github.com/enso-org/ide/releases/latest).

Dear HN Family, we are super excited to show Enso to you. Please, share with us your thoughts, experiences, ideas and feedback. It is insanely important to us, as our dream is to make Enso the most useful data processing platform in your toolbox! Also, in case you’d like to build your projects on top of Enso, we would love to help you do it – describe what you have in mind here, and we will reach out to you: https://airtable.com/shrsnx2mJuRn0MxIS :)

=== Links ===

[1] Luna: Visual and textual functional programming language* - https://news.ycombinator.com/item?id=11144828 - Feb 2016 (100 comments)

Enso the language is mostly scala, Enso the IDE, which is very much an integral part of the project, is like 90% Rust.

The whole IDE (cloud + desktop one) is written in Rust. It lives in this repo: https://github.com/enso-org/ide :)

These issues with random zoom in/out or with selection problems are not known. We haven't seen them before. Would you be so nice to create a screencast and post it as an issue / issues on our issue tracker? This would allow us to track it and fix it for the next release: https://github.com/enso-org/ide/issues .

Hi, Wojciech, one of Enso founders here! We are just preparing for Enso 2.0 release. If you want to play with it, you can download the current build from our GitHub releases page (https://github.com/enso-org/ide/releases) and see intro tutorials here: https://www.youtube.com/channel/UC4oMK7cL1ElfNR_OhS-YQAw

:)

I'd use a modeling tool like https://mtk.sciml.ai/dev/ Using the standard library, you wouldn't need to come up with all equations yourself. Depending on the details of your use case, system identification as suggested before might be a faster approach though.

No, you do get type errors during runtime. The most common one is a MethodNotFound error, which corresponds to a dispatch not being found. This is the one that people then complain about for long stacktraces and as being hard to read (and that's a valid criticism). The reason for it is because if you do xy with a type combination that does not have a corresponding dispatch, i.e. (x::T1,y::T2) not defined anywhere, then it looks through the method table of the function, does not find one, and throws this MethodNotFound error. You will only get no error if a method is found. Now what can happen is that you can have a method to an abstract type, *(x::T1,y::AbstractArray), but `y` does not "actually" act like an AbstractArray in some way. If the way that it's "not an AbstractArray" is that it's missing some method overloads of the AbstractArray interface (https://docs.julialang.org/en/v1/manual/interfaces/#man-inte...), you will get a MethodNotFound error thrown on that interface function. Thus you will only not get an error if someone has declared `typeof(y) <: AbstractArray` and implemented the AbstractArray interface.

However, what Yuri pointed out is that there are some packages (specifically in the statistics area) which implemented functions like `f(A::AbstractArray)` but used `for i in 1:length(A)` to iterate through x's values. Notice that the AbstractArray interface has interface functions for "non-traditional indices", including `axes(A)` which is a function to call to get "the a tuple of AbstractUnitRange{<:Integer} of valid indices". Thus these codes are incorrect, because by the definition of the interface you should be doing `for i in axes(A)` if you want to support an AbstractArray because there is no guarantee that its indices go from `1:length(A)`. Note that this was added to the `AbstractArray` interface in the v1.0 change, which is notably after the codes he referenced were written, and thus it's more that they were not updated to handle this expanded interface when the v1.0 transition occurred.

This is important to understand because the criticisms and proposed "solutions" don't actually match the case... at all. This is not a case of Julia just letting anything through: someone had to purposefully define these functions for them to exist. And interfaces are not a solution here because there is an interface here, its rules were just not followed. I don't know of an interface system which would actually throw an error if someone does a loop `for i in 1:length(A)` in a code where `A` is then indexed by the element. That analysis is rather difficult at the compiler level because it's non-local: `length(A)` is valid since querying for the length is part of the AbstractArray interface (for good reasons), so then `1:length(A)` is valid since that's just range construction on integers, so the for loop construction itself is valid, and it's only invalid because of some other knowledge about how `A[i]` should work (this look structure could be correct if it's not used to `A[i]` but rather do something like `sum(i)` without indexing). If you want this to throw an error, the only real thing you could do is remove indexing from the AbstractArray interface and solely rely on iteration, which I'm not opposed to (given the relationship to GPUs of course), but etc. you can see the question to solving this is "what is the right interface?" not "are there even interfaces?" (of which the answer is, yes but the errors are thrown at runtime MethodNotFound instead of compile time MethodNotImplemented for undefined things, the latter would be cool for better debugging and stacktraces but isn't a solution).

This is why the real discussions are not about interfaces as a solution, they don't solve this issue, and even further languages with interfaces also have this issue. It's about tools for helping code style. You probably should just never do `for i in 1:length(A)`, probably you should always do `for i in eachindex(A)` or `for i in axes(A)` because those iteration styles work for `Array` but also work for any `AbstractArray` and thus it's just a safer way to code. That is why there are specific mentions to not do this in style guides (for example, https://github.com/SciML/SciMLStyle#generic-code-is-preferre...), and things like JuliaFormatter automatically flag it as a style break (which would cause CI failures in organizations like SciML which enforce SciML Style formatting as a CI run with Github Actions https://github.com/SciML/ModelingToolkit.jl/blob/v8.14.1/.gi...). There's a call to add linting support for this as well, flagging it any time someone writes this code. If everyone is told to not assume 1-based indexing, formatting CI fails if it is assumed, and the linter underlines every piece of code that does it as red, (along with many other measures, which includes extensive downstream testing, fuzzing against other array types, etc.) then we're at least pretty well guarded against it. And many Julia organizations, like SciML, have these practices in place to guard against it. Yuri's specific discussion is more that JuliaStats does not.

The thing to watch in the space of Simulink/Modelica is https://github.com/SciML/ModelingToolkit.jl . It's an acausal modeling system similar to Modelica (though extended to things like SDEs, PDEs, and nonlinear optimization), and has a standard library (https://github.com/SciML/ModelingToolkitStandardLibrary.jl) similar to the MSL. There's still a lot to do, but it's pretty functional at this point. The two other projects to watch are FunctionalModels.jl (https://github.com/tshort/FunctionalModels.jl, which is the renamed Sims.jl), which is built using ModelingToolkit.jl and puts a more functional interface on it. Then there's Modia.jl (https://github.com/ModiaSim/Modia.jl) which had a complete rewrite not too long ago, and in its new form it's fairly similar to ModelingToolkit.jl and the differences are more in the details. For causal modeling similar to Simulink, there's Causal.jl (https://github.com/zekeriyasari/Causal.jl) which is fairly feature-complete, though I think a lot of people these days are going towards acausal modeling instead so flipping Simulink -> acausal, and in that transition picking up Julia, is what I think is the most likely direction (and given MTK has gotten 40,000 downloads in the last year, I think there's good data backing it up).

And quick mention to bring it back to the main thread here, the DataDrivenDiffEq symbolic regression API gives back Symbolics.jl/ModelingToolkit.jl objects, meaning that the learned equations can be put directly into the simulation tools or composed with other physical models. We're really trying to marry this process modeling and engineering world with these "newer" AI tools.

In this case, yes, this should just be done numerically. But using symbolic transformations to optimize numeric code is also a really neat application of symbolic computing that doesn't get enough attention, imo. [This library](https://github.com/SciML/ModelingToolkit.jl), for example, uses symbolics to do sparsity detection, automatic derivative/gradient/jacobian/hessian calculations, index reduction, etc. to speed up numerical differential equation solving.

https://homes.cs.washington.edu/~thickstn/ctpg-project-page/...

That's all showing the raw iteration count to show that it algorithmically is faster, but the time per iteration is also fast for many reasons showcased in the SciMLBenchmarks routinely outperforming C and Fortran solvers (https://github.com/SciML/SciMLBenchmarks.jl). So it's excelling pretty well, and things like the automated discovery of black hole dynamics are all done using the universal differential equation framework enabled by the SciML tools (see https://arxiv.org/abs/2102.12695 for that application).

What we are missing however is that, right now these simulations are all writing raw differential equations so we do need a better set of modeling tools. That said, MuJoCo and DiffTaichi are not great physical modeling environments for building real systems, instead we would point to Simulink and Modelica as what are really useful for building real-world systems. So it would be cool if there was a modeling language in Julia which extends that universe and directly does optimal code generation for the Julia solvers... and that's what ModelingToolkit.jl is (https://github.com/SciML/ModelingToolkit.jl). That project is still pretty new, but there's already enough to show some large-scale models outperforming Dymola on examples that require symbolic tearing and index reduction, which is far more than what physical simulation environments used for non-scientific purposes (MuJoCo and DiffTaichi) are able to do. See the workshop for details (https://www.youtube.com/watch?v=HEVOgSLBzWA). And that's just the top level details, there's a whole Julia Computing product called JuliaSim (https://juliacomputing.com/products/juliasim/) which is then being built on these pieces to do things like automatically generate ML-accelerated components and add model building GUIs.

That said, MuJoCo and DiffTaichi have much better visualizations and animations than MTK. Our focus so far has been on the core routines, making them fast, scalable, stable, and extensive. You'll need to wait for the near future (or build something with Makie) if you want the pretty pictures of the robot to happen automatically. That said, Julia's Makie visualization system has already been shown to be sufficiently powerful for this kind of application (https://nextjournal.com/sdanisch/taking-your-robot-for-a-wal...), so we're excited to see where that will go in the future.

PDEs are difficult because you don't have a simple numerical definition over all PDEs because they can be defined by arbitrarily many functions. u' = Laplace u + f? Define f. u' = g(u) * Laplace u + f? Define f and g. Etc. To cover the space of PDEs you have to go symbolic at some point, and make the discretization methods dependent on the symbolic form. This is precisely what the ModelingToolkit.jl ecosystem is doing. One instantiation of a discretizer on this symbolic form is NeuralPDE.jl which takes a symbolic PDESystem and generates an OptimizationProblem for a neural network which represents the solution via a Physics-Informed Neural Network (PINN).

There's a very clear momentum for Julia here in this domain of modeling and simulation. With JuliaSim funding an entire modeling and simulation department within Julia Computing dedicated to building out an ecosystem that accelerates this domain and the centralization around the SciML tooling, this is an area where we absolutely have both a manpower and momentum advantage. We're getting many universities (PhD students and professors) involved on the open source side, while building out different commercial tools and GUIs on top of the open numerical core. The modeling and simulation domain itself is soon going to have its own SciMLCon since our developer community has gotten too large to just be a few JuliaCon talks: it needs its own days to fit everyone! Not only that, in many aspects we're not just moving faster but have already passed. Not in every way, there's still some important discussion in controls that needs to happen, but that's what the momentum is for.

Julia has great support for modeling, have a look at ModelingToolkit.jl. From the README: