frameless VS Laminar

I don't see how Spark is any "less controversial" when the Spark Delay instance for cats-effect takes an entire SparkSession implicitly.

To add to this I lately have used Spark with frameless for compile time safety and it's an interesting library that works well with Spark.

For type safety with dataframes, techniques like https://github.com/typelevel/frameless can be used.

The preferred way to write Spark programs is to use DataFrame API which is untyped and is essentially the same in Scala, C# and Python. It's a DSL that's used to describe AST of the computation and the end result is the same regardless of language. There's a library called Frameless (https://github.com/typelevel/frameless) that implements typed DataFrame API but it is not in wide use, it looked dead for quite some time (though now development seems to continue) and didn't play nice with IntelliJ IDEA last time I checked. Performance-wise there's no difference most of the time (since all the program does is create an AST) except when using UDFs - Python UDFs are significantly slower and you can't write "proper" UDFs in Python - ones that generate Java code.

Frameless is a nice way to grab some type safety back from Spark, and features opt-in Cats integration.

Valid point! Have you seen the withColumnTupled API? It returns a typed tuple instead. This seems to satisfy your use case - the dataset preserves its type and doesn't require a new case class. This is kind of what you're suggesting but without case class generation. Though not sure whether attribute labels (names) are preserved in this case. It's also unclear whether this is good enough for wide tables.

I recommend using Frameless, which includes a Cats module. In general, I would encourage you to master “purely” functional programming first, because it’s foundational. Spark is a very specific technology, and probably not even the best in that class today—I would be very careful about trying to build a career around it.

My quite niche open source project broke this threshold last year, via Github sponsorships. Of course, I put a lot of time into it, so it's not "passive income" or even "market rate income", but still, without these sponsorships I wouldn't be able to work on it so much.

The project is Laminar, a UI library for Scala.js https://laminar.dev

I've noticed that Laminar and the newly released Kyo point that they don't use a lot of macros as a feature. Laminar says "Easy to understand: no macros", while Kyo emphasizes "Note: defer is currently the only macro in Kyo. All other features use regular language constructs." It seems that using less macros is something library users will like.

https://laminar.dev/ might be what you need. Though I wish there was a more beginner friendly (I'm not from front-end world) tutorial for me to follow along.

Have you looked at Laminar and Tyrian? Especially Tyrian seems to be close to what you're looking for.

For Scala there is Laminar, which has an even flashier website with nice docs. I haven't tested it out though, as I have never used Scala.

I found this scala library called Laminar which looks super similar to solid. They use signals and has no virtual dom. State changes are represented by signals and events by event streams. Thus they seems to have feature parity with RXJS as they can model all sorts of async stuff. Best part is they get to keep writing their markup in C-style syntax than XML based JSX. It looks super elegant,minimalist and has type safety.

This is very true. I really hate svelte single file components. But then I tried JSX for breaking things down. I love solid but I don't feel really good about angle brackets within C style syntax. I saw this Scala library that stick with simple statically typed function syntax than html tags. I don't understand why people still wants to stick with xml like tags. In laminar markup is written like this scala div( h1("Hello world", color := "red"), inputCaption, input(inputMods, name := "fullName"), div( ">>", button("Submit"), "<<" ) ) I wish solid team makes their HyperScript syntax as performant as JSX.

Laminar (Scala framework) hasn't been mentioned yet so dropping it here as an awesome framework that support HTML-over-the-wire. It can be used together with React, HTMX, and many other frontend frameworks -- but doesn't have to be.

https://laminar.dev/

Scala.js core itself, which I maintain, does not need much innovation. We support all of Scala, and interact with any JavaScript library. That's what the core promises.

If you want to compare to Scala 3, it's worth pointing out that you can use Scala.js with any Scala version >= 2.12.2. In particular, you can use it with Scala 3 and benefit from all its innovations. ;)

Innovation comes mainly from libraries, notably UI libraries. Laminar (https://laminar.dev/) is a great example.

In terms of roadmap, we are mostly working on "boring" stuff: improving performance (of the generated code, and of the linker), fixing bugs when they get reported, etc.

Perhaps, when Wasm gets more features for deeper interoperability with JavaScript (manipulating objects notably), we will take another look at targeting Wasm. People usually expect all languages to target Wasm now, "because it's fast". Truth is, it's fast for languages with linear memory. There is no evidence yet that it will be fast for memory-managed languages with objects and virtual dispatch.