kotlin-spark-api VS Slick

https://github.com/Kotlin/kotlin-spark-api this definitely needs to be in the language bindings section 😅

Kotlin should mostly have no problems being called in Scala. Give kotlin-spark-api a look to figure out how would that work the other way around.

Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, and better Java interop.

https://github.com/JetBrains/kotlin-spark-api

Made a PR on slick to document this https://github.com/slick/slick/pull/2760 (workaround is quite easy, you can just define def tupled = (apply _).tupled in the companion object of the case class and it will also compile for all Scala versions).

The Scala ecosystem has a few ways to do composable type-safe query building, e.g. Slick[0] or more recently Quill[1]. . I believe both also have ways to do compile-time string interpolation (e.g. sql"""select * from users where id = ${user.id}""") which generate prepared statements (I know Slick does prepared statements. Quill has similar macros but I haven't looked into how safe they are to use).

[0] https://scala-slick.org/

Release notes at https://github.com/slick/slick/releases/tag/v3.5.0-M3

With all this in mind, I landed at the first candidate: slick from https://scala-slick.org/ that you all probably know.

I think Slick's current priority is also getting in Scala 3 support: https://github.com/slick/slick/issues/2177

Future releases might not be announced here. To get notified, go to https://github.com/slick/slick, click the Watch dropdown button at the top, select Custom, check Releases, and click Apply.

https://github.com/slick/slick/pull/2097 now I use slightly lower version of slick so this might be an upgrade that resolves (I do recall using it in 21 and it was still buggy and I filed a ticket, which I cannot find at the moment), but given a complex enough query (we have one in PROD which has tons of flexibility in terms of filters that can be passed in) but it also makes for complex code.

I started writing a reply but then I realized it would be long and depends on exactly what you mean, so maybe it's better to post the question in https://github.com/slick/slick/discussions/categories/questions?

I purchased the very entertaining book Seven Languages in Seven Weeks. Although I found Haskell fascinating and tempting, I knew it was unrealistic to introduce it in our company. Scala on the other hand looked like it could be the holy grail: All the characteristics I was looking for, no need to abandon the JVM and its cornucopia of tools and libraries, and the possibility for coexistence with Java and therefore incremental adoption. After implementing some simple programs to identify any immediate risks of committing to the language and its ecosystem, I started to introduce Scala in customer projects. Luckily, I was fortunate enough to work with open-minded, curious, and ambitious team members who were also experienced enough to appreciate the benefits of the language. We immediately applied our experience with functional programming, and embraced immutability. Libraries like Slick and Akka HTTP (we actually started out with its predecessor, Spray) made building database-backed REST services a breeze. And the resulting code was robust and highly maintainable. Scala's expressive type system and type inference made it easy to build a restrictive, consistent domain model without bloating the code. There was virtually no overhead. Any boilerplate could be easily abstracted out. In the end, the application code felt natural, concise and elegant. Programming was fun again.