ide VS PackageCompiler.jl

docs (https://enso.org/docs/syntax). We've added many new libraries, so you can do many more things with it now. Oh, and we changed the name to Enso and got accepted to YC! :)

The problem we address is that data analysts still waste up to half of their time on repetitive manual work that can be automated [6]. To give one example, a company we're working with hires business users who use Excel to define data quality rules. These get manually translated to SQL, then manually translated to Python. This is not only error prone, it’s so slow that it takes them 90 days to introduce a single new rule. There’s 60 days’ worth of overhead in this process—it’s insane!

Years ago I (Wojciech) led the in-house development of visual effects (VFX) tools at a motion picture studio. We made tools like cloud renderers and smoke simulation engines. The artists using these tools did not have any programming background, yet they were designing complex algorithms for forces between particles, light subsurface scattering, things like that. Earlier generations of these tools had hundreds of config options, buttons, etc., for masses of different use cases, but this approach got way too complex and people eventually realized that it falls short when you need to do anything that the vendor did not think of. Nowadays they use node-based software (like the Houdini FX) which lets users draw algorithms as a sequence of data processing steps (these steps are often referred to as “nodes”). Later, when I was working in other industries and encountered the same rats’ nests of complex GUIs for solving data processing problems, I realized that the data analytics/science space was in need of the same breakthrough that we had already gone through in the VFX space.

Most visual programming languages / workflow-builders do not scale well because they don't let users express abstractions. Try to build a complex pipeline and you'll end with an unreadable spaghetti of connections—it's like coding a web app in the assembler. Enso is different because we allow you to build abstractions to manage the complexity. As a result, you never have more than 10-20 nodes on the stage in Enso (nodes are hierarchical). You can create custom data types, custom components (functions), catch errors, etc. All this works because under the hood, Enso is a real programming language. However, naive implementations of such systems are super slow. Each component may be built of hundreds, sometimes thousands of lower-level ones. The real trick is making these hierarchical components run fast. For that you need a dedicated compiler and a runtime system, and this is a hard technical space. Our system involves a dedicated JIT compiler based on GraalVM. For details, see https://enso.org/language#compiler. In case this is interesting for you, here is our podcast about how the compiler works under the hood: https://www.youtube.com/watch?v=BibjcUjdkO4.

Enso is interactive, meaning that we recompute the relevant parts of graphs as parameters change, which shortens feedback loops dramatically. Like a lot of people on HN, we were inspired by Bret Victor's classic talk on instant feedback: https://www.youtube.com/watch?v=8QiPFmIMxFc. We’ve also put a lot of effort into extensibility. You can add Java, JavaScript, R, and Python (soon also Ruby, Scala, Kotlin, Rust, and C) directly into Enso nodes without the need to write any wrappers and with a close-to-zero performance overhead.

Enso is open source. Our compiler code is at https://github.com/enso-org/enso and our GUI code at https://github.com/enso-org/ide. Our business model is based on selling domain specific libraries, on-premise installations with enhanced user permission management, and coming soon, a hosted solution called Enso Cloud, which will be our only non-open-source codebase. Since this is Hacker News, I should add that all our alpha releases collect anonymous usage statistics which we use to improve Enso and prepare it for a stable release. Full details about that are always in our release notes (https://github.com/enso-org/ide/releases/latest).

Dear HN Family, we are super excited to show Enso to you. Please, share with us your thoughts, experiences, ideas and feedback. It is insanely important to us, as our dream is to make Enso the most useful data processing platform in your toolbox! Also, in case you’d like to build your projects on top of Enso, we would love to help you do it – describe what you have in mind here, and we will reach out to you: https://airtable.com/shrsnx2mJuRn0MxIS :)

=== Links ===

[1] Luna: Visual and textual functional programming language* - https://news.ycombinator.com/item?id=11144828 - Feb 2016 (100 comments)

Enso the language is mostly scala, Enso the IDE, which is very much an integral part of the project, is like 90% Rust.

The whole IDE (cloud + desktop one) is written in Rust. It lives in this repo: https://github.com/enso-org/ide :)

These issues with random zoom in/out or with selection problems are not known. We haven't seen them before. Would you be so nice to create a screencast and post it as an issue / issues on our issue tracker? This would allow us to track it and fix it for the next release: https://github.com/enso-org/ide/issues .

Hi, Wojciech, one of Enso founders here! We are just preparing for Enso 2.0 release. If you want to play with it, you can download the current build from our GitHub releases page (https://github.com/enso-org/ide/releases) and see intro tutorials here: https://www.youtube.com/channel/UC4oMK7cL1ElfNR_OhS-YQAw

:)

Yes, julia can be called from other languages rather easily, Julia functions can be exposed and called with a C-like ABI [1], and then there's also various packages for languages like Python [2] or R [3] to call Julia code.

With PackageCompiler.jl [4] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed.

Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [5] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype)

[1] https://docs.julialang.org/en/v1/manual/embedding/

[2] https://pypi.org/project/juliacall/

[3] https://www.rdocumentation.org/packages/JuliaCall/

[4] https://github.com/JuliaLang/PackageCompiler.jl

[5] https://github.com/tshort/StaticCompiler.jl

One of Julia's Achilles heels is standalone, ahead-of-time compilation. Technically this is already possible [1], [2], but there are quite a few limitations when doing this (e.g. "Hello world" is 150 MB [7]) and it's not an easy or natural process.

The immature AoT capabilities are a huge pain to deal with when writing large code packages or even when trying to make command line applications. Things have to be recompiled each time the Julia runtime is shut down. The current strategy in the community to get around this seems to be "keep the REPL alive as long as possible" [3][4][5][6], but this isn't a viable option for all use cases.

Until Julia has better AoT compilation support, it's going to be very difficult to develop large scale programs with it. Version 1.9 has better support for caching compiled code, but I really wish there were better options for AoT compiling small, static, standalone executables and libraries.

[1]: https://julialang.github.io/PackageCompiler.jl/dev/

Doesn’t work on Windows, but https://github.com/JuliaLang/PackageCompiler.jl does.

Also, you can precompile a whole package and just ship the binary. We do this all of the time.

https://github.com/JuliaLang/PackageCompiler.jl

And getting things precompiled: https://sciml.ai/news/2022/09/21/compile_time/

You can have a look at PackageCompiler.jl

I think by PackageManager here you mean package compiler, and yes these improvements do not need a 2.0. v1.8 included a few things to in the near future allow for building binaries without big dependencies like LLVM, and finishing this work is indeed slated for the v1.x releases. Saying "we are not doing a 2.0" is precisely saying that this is more important than things which change the user-facing language semantics.

And TTFP does need to be addressed. It's a current shortcoming of the compiler that native and LLVM code is not cached during the precompilation stages. If such code is able to precompile into binaries, then startup time would be dramatically decreased because then a lot of package code would no longer have to JIT compile. Tim Holy and Valentin Churavy gave a nice talk at JuliaCon 2022 about the current progress of making this work: https://www.youtube.com/watch?v=GnsONc9DYg0 .

This is all tied up with startup time and are all in some sense the same issue. Currently, the only way to get LLVM code cached, and thus startup time essentially eliminated, is to build it into what's called the "system image". That system image is the binary that package compiler builds (https://github.com/JuliaLang/PackageCompiler.jl). Julia then ships with a default system image that includes the standard library in order to remove the major chunk of code that "most" libraries share, which is why all of Julia Base works without JIT lag. However, that means everyone wants to have their thing, be it sparse matrices to statistics, in the standard library so that it gets the JIT-lag free build by default. This means the system image is huge, which is why PackageCompiler, which is simply a system for building binaries by appending package code to the system image, builds big binaries. What needs to happen is for packages to be able to precompile in a way that then caches LLVM and native code. Then there's no major compile time advantage to being in the system image, which will allow things to be pulled out of the system image to have a leaner Julia Base build without major drawbacks, which would then help make the system compile. That will then make it so that an LLVM and BLAS build does not have to be in every binary (which is what takes up most of the space and RAM), which would then allow Julia to much more comfortably move beyond the niche of scientific computing.

You're probably dancing around the edges of what [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl) is capable of targeting. There are a few new capabilities coming online, namely [separating codegen from runtime](https://github.com/JuliaLang/julia/pull/41936) and [compiling small static binaries](https://github.com/tshort/StaticCompiler.jl), but you're likely to hit some snags on the bleeding edge.