TidyverseSkeptic VS PackageCompiler.jl

I just don't get these to be honest -- besides the fact that author missed simple things like `df.groupby('var',as_index=False)`, isn't this obviously arbitrary "this is easier my way" complaints? (I did R before all the chaining stuff was popular, and I wouldn't stuff everything into a single command like that even now. It isn't like you get lazy evaluation or any special data processing magic.)

So I get people love chaining and tidyverse, good for you, I don't. But at least I can acknowledge that my way (or this way) people have different preferences and one is not intrinsically easier.

Norm Matloff has a blog where he essentially just argues the opposite of all the tidyverse stuff, https://github.com/matloff/TidyverseSkeptic, but it is the same idea in reverse to me (one is not obviously easier to learn than the other IMO).

On the other hand, there is also a more traditional universe outside of the of the newer tidyverse approach. See the criticism of the tidyverse ecosystem by Prof Norm Matloff (of UC Davis). He provides a freely available introductory course in base R.

Whenever I hear tidyverse, I just feel the need to leave this: TidyverseSceptic

Yeah, I had never heard of him before, but I followed the link in the article above to his GitHub page and think he made some really great points about conciseness and clarity in base R code, and, I admittedly had no idea tapply() was so useful and easy to use, because I almost never see it used in any examples online. Although I agree with others here that he's misrepresenting why package developers use base R (which is to avoid dependences in their packages, which is very important), I also find myself agreeing with him that future R programmers not being taught base R is worrisome (I'm thinking of dependencies in future package development).

I was in an R group meeting. One of the members mentioned Prof. Norm Matloff and said he has comments about tidyverse. I searched and found Matloff's explanation here. What are your thoughts on tidyverse and Matloff's comments about it? As I read it, I found myself agreeing with certain points. I do not have a computer science background; I'm someone trying to learn coding because I see uses for it in my work. I started my learning, about a year ago, with tidyverse tutorials. My patchwork jumping around, maybe in addition to some of the gaps Matloff indicates, show me that I know very little about base R.

There are good arguments against tidyverse, especially for beginners. It doesn't lead to a growth in understanding the language fundamentals and requires to learn many functions, paradigms, and syntaxes not shared by base R, which can easily be overwhelming and lead to a learn-by-heart approach more than to a learn-by-understanding. There are many good articles on the topic, such as this one or a more in-depth one, suggesting to consider tidyverse a more advanced application for specific use cases, if you like the dialect. I don't, so I might be biased.

I would discourage you to get into the tidyverse, at least in the first stages of your R training. It's like trying to learn english AND scottish together as a foreigner. You can read some better worded discussions here https://github.com/matloff/TidyverseSkeptic and here https://towardsdatascience.com/a-thousand-gadgets-my-thoughts-on-the-r-tidyverse-2441d8504433?gi=1b0a3648b6e6

My opinions are largely in agreement with Norm Matloff on the subject actually.

Yes, julia can be called from other languages rather easily, Julia functions can be exposed and called with a C-like ABI [1], and then there's also various packages for languages like Python [2] or R [3] to call Julia code.

With PackageCompiler.jl [4] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed.

Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [5] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype)

[1] https://docs.julialang.org/en/v1/manual/embedding/

[2] https://pypi.org/project/juliacall/

[3] https://www.rdocumentation.org/packages/JuliaCall/

[4] https://github.com/JuliaLang/PackageCompiler.jl

[5] https://github.com/tshort/StaticCompiler.jl

One of Julia's Achilles heels is standalone, ahead-of-time compilation. Technically this is already possible [1], [2], but there are quite a few limitations when doing this (e.g. "Hello world" is 150 MB [7]) and it's not an easy or natural process.

The immature AoT capabilities are a huge pain to deal with when writing large code packages or even when trying to make command line applications. Things have to be recompiled each time the Julia runtime is shut down. The current strategy in the community to get around this seems to be "keep the REPL alive as long as possible" [3][4][5][6], but this isn't a viable option for all use cases.

Until Julia has better AoT compilation support, it's going to be very difficult to develop large scale programs with it. Version 1.9 has better support for caching compiled code, but I really wish there were better options for AoT compiling small, static, standalone executables and libraries.

[1]: https://julialang.github.io/PackageCompiler.jl/dev/

Doesn’t work on Windows, but https://github.com/JuliaLang/PackageCompiler.jl does.

Also, you can precompile a whole package and just ship the binary. We do this all of the time.

https://github.com/JuliaLang/PackageCompiler.jl

And getting things precompiled: https://sciml.ai/news/2022/09/21/compile_time/

You can have a look at PackageCompiler.jl

I think by PackageManager here you mean package compiler, and yes these improvements do not need a 2.0. v1.8 included a few things to in the near future allow for building binaries without big dependencies like LLVM, and finishing this work is indeed slated for the v1.x releases. Saying "we are not doing a 2.0" is precisely saying that this is more important than things which change the user-facing language semantics.

And TTFP does need to be addressed. It's a current shortcoming of the compiler that native and LLVM code is not cached during the precompilation stages. If such code is able to precompile into binaries, then startup time would be dramatically decreased because then a lot of package code would no longer have to JIT compile. Tim Holy and Valentin Churavy gave a nice talk at JuliaCon 2022 about the current progress of making this work: https://www.youtube.com/watch?v=GnsONc9DYg0 .

This is all tied up with startup time and are all in some sense the same issue. Currently, the only way to get LLVM code cached, and thus startup time essentially eliminated, is to build it into what's called the "system image". That system image is the binary that package compiler builds (https://github.com/JuliaLang/PackageCompiler.jl). Julia then ships with a default system image that includes the standard library in order to remove the major chunk of code that "most" libraries share, which is why all of Julia Base works without JIT lag. However, that means everyone wants to have their thing, be it sparse matrices to statistics, in the standard library so that it gets the JIT-lag free build by default. This means the system image is huge, which is why PackageCompiler, which is simply a system for building binaries by appending package code to the system image, builds big binaries. What needs to happen is for packages to be able to precompile in a way that then caches LLVM and native code. Then there's no major compile time advantage to being in the system image, which will allow things to be pulled out of the system image to have a leaner Julia Base build without major drawbacks, which would then help make the system compile. That will then make it so that an LLVM and BLAS build does not have to be in every binary (which is what takes up most of the space and RAM), which would then allow Julia to much more comfortably move beyond the niche of scientific computing.

You're probably dancing around the edges of what [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl) is capable of targeting. There are a few new capabilities coming online, namely [separating codegen from runtime](https://github.com/JuliaLang/julia/pull/41936) and [compiling small static binaries](https://github.com/tshort/StaticCompiler.jl), but you're likely to hit some snags on the bleeding edge.