spark-excel
Quill
spark-excel | Quill | |
---|---|---|
8 | 15 | |
439 | 2,136 | |
1.6% | 0.0% | |
8.6 | 9.0 | |
6 days ago | 7 days ago | |
Scala | Scala | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
spark-excel
- Pandas was faster and less memory intensive then crealytics pyspark. How is it possible?
-
Automating Excel to Databricks Table
Not natively. But the com.crealytics.spark.excel library has had great results for us. There are still some cases where pandas manipulation is needed with Excel files that have weird header setups.
-
Can AWS Glue convert a JSON payload to Excel tab? (not csv)
Spark 3.2 has a to_excel() method, but not Spark 3.1, so you'll need to use an external library such as https://github.com/crealytics/spark-excel
-
Reading a xlsx file with PySpark
Have you checked spark-excel's documentation? The dataAddress option seems to be what you're looking for.
-
read percentage values in spark ( no casting )
Are you using this library to load xlsx files? https://github.com/crealytics/spark-excel
-
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.executor.memory;
I found similiar issues on their github: https://github.com/crealytics/spark-excel/issues/227
-
How do I learn to read a plug-in?
Plug-in in question is GitHub - crealytics/spark-excel: A Spark plugin for reading Excel files via Apache POI , but I guess it could be any. Assuming that I can read the plain code in an individual .scala file how do I learn to understand how it all pieces together and what the underlying code being run is?
Quill
-
Dear Sir, You Have Built a Compiler (2022)
https://github.com/zio/zio-quill
This library does exactly what you prescribe. Pretty sure under the hood it's using macros with string templates
- Sketch of a Post-ORM
-
Why use Spark?
But I can connect to Postgress with something like Quill and run sophisticated queries to fetch data. Which then got me thinking, what is the difference between using Spark to connect to the database and using something like Quill or your normal pure JDBC driver?
-
What's the point of opaque type aliases (and are they actually sound)?
Just as an example, say you are using quill ( https://getquill.io/ ) to query your database.
- I want to move to Scala 3, but I'm not sure what libraries to use
-
Query DSL in Scala ?
I think Quill is the closest to your request: https://github.com/zio/zio-quill
-
Doobie tutorial: databases and pure FP in Scala
If this still looks like too much hassle, you can always go a bit higher-level and use something like Quill, which is also a powerful approach that uses a different, more ORM-like style.
-
Ask HN: What cutting-edge technology do you use?
I'm using it mostly for full-stack web development with ScalaJS (https://www.scala-js.org) in the frontend (https://outwatch.github.io/docs/readme.html) and in the backend with AWS lambdas.
The ecosystem is currently in the process of porting all the libraries to Scala 3. So if you're new to Scala, I'd recommend to start with Scala 2, which is rock-solid and already very powerful.
I never worked with SQLAlchemy. But on the scala database side, popular libraries are Doobie (https://tpolecat.github.io/doobie) and Quill (https://getquill.io). Keep in mind that these are for Scala on the JVM. On the ScalaJS side I'm using the javascript library pg. But I'd like to try if it works well with Prisma soon.
The nice thing about ScalaJS is, that you can use Javascript libraries. And if there are typescript facades, then you can transpile these to Scala and use them in a type safe way (https://scalablytyped.org).
-
Fp libraries that target scala 3 exclusively?
I know that libraries like Scodec and shapeless were rewritten practically from scratch for Scala 3, taking advantage of the next syntax and internals, as well as protoquill - a Scala 3 implementation of Quill.
-
Best Scala framework / libraries out there ?
Akka HTTP, Cats, Quill, ninny, Monix Observable, mill.