chispa
rupy
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
chispa
-
Spark open source community is awesome
here's a little README fix a user pushed to chispa
-
Invitation to collaborate on open source PySpark projects
chispa is a library of PySpark testing functions.
-
installing pyspark on my m1 mac, getting an env error
The other approach I've used is Poetry, see the chispa project as an example. Poetry is especially nice for projects that you'd like to publish to PyPi because those commands are built-in.
-
Spark: local dev environment
- All Spark transformations are tested with pytest + chispa (https://github.com/MrPowers/chispa)
-
Pyspark now provides a native Pandas API
Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.
-
Show dataengineering: beavis, a library for unit testing Pandas/Dask code
I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.
-
Tips for building popular open source data engineering projects
Blogging has been the main way I've been able to attract users. Someone searches "testing PySpark", they see this blog, and then they're motivated to try chispa.
-
Ask HN: What are some tools / libraries you built yourself?
I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.
quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.
Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.
Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8
-
Open source contributions for a Data Engineer?
I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.
-
Why Databricks Is Winning
The last point was for teams that only rely on notebooks, sorry if I didn't make that clear.
You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files.
Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6
rupy
-
Considerations for a long-running Raspberry Pi
I have been running a Raspberry 2 cluster for 10 years: http://host.rupy.se
A few weeks back the first SD card to fail got so corrupted it failed to reboot!
My key learning is use oversized cards, because then the bitcycle will wear slower!
I'm going from 32GB to 256/512/1024!
- Sandstorm: Open-source platform for self-hosting web app
-
You Want Modules, Not Microservices
I think we're all confused over the definition. Also one might understand what all the proponents are talking about better if they think about this more as a process and not some technological solution:
https://github.com/tinspin/rupy/wiki/Process
All input I have is you want your code to run on many machines, in fact you want it to run the same on all machines you need to deliver and preferably more. Vertically and horizontally at the same time, so your services only call localhost but in many separate places.
This in turn mandates a distributed database. And later you discover it has to be capable of async-to-async = no blocking ever anywhere in the whole solution.
The way I do this is I hot-deploy my applications async. to all servers in the cluster, this is what a cluster node looks like in practice (the name next to Host: is the node): http://host.rupy.se if you click "api & metrics" you'll see the services.
With this not only do you get scalability, but also redundancy and development is maintained at live coding levels.
-
I wish my web server were in the corner of my room
I have hosted my own web server both physically and codevise since 2014.
It's on a Raspberry 2 cluster:
Since 2016 i have my own database also coded from scratch:
We need to implement HTTP/1.1 with less bloat, a C non-blocking web server that can share memory between threads is probably the most interesting project for humans right now, is anyone working on that?
-
Ask HN: Free and open source distributed database written in C++ or C
I have one in Java: https://github.com/tinspin/rupy
Here is the 2000 lines of code of the entire database: http://root.rupy.se/code?path=/Root.java
And here you can try it out: http://root.rupy.se
-
Dokku – Free Heroku Alternative
The smallest PaaS you have ever seen is one order of magnitude larger than mine: https://github.com/tinspin/rupy
And I bet you the same goes for performance, if not two!
-
Server-Sent Events: the alternative to WebSockets you should be using
Absolutely not, HTTP/1.1 is the way to make SSE fly:
https://github.com/tinspin/rupy/wiki/Comet-Stream
Old page search for "event-stream"... Comet-stream is a collection of techniques of which SSE is one. My findings are that SSE go through anti-viruses better!
I would look at my own app-server: https://github.com/tinspin/rupy
It's not the most well documented but it's the smallest implementation while still being one of the most performant so you can learn more than just SSE.
The data is here: http://fuse.rupy.se/about.html
Under Performance. Per watt the fuse/rupy platform completely crushes all competition because of 2 reasons:
- Event driven protocol design, averages at about 4 messages/player/second (means you cannot do spraying or headshots f.ex. which is another feature in my game design opinion).
- Java's memory model with atomic concurrency which needs a VM and GC (C++ copied that memory model in C++11, but it failed completely because they lack both VM and GC, but that model is still to this day the one C++ uses), you can read more about this here: https://github.com/tinspin/rupy/wiki
You can argue those points are bad arguments, but if you look at performance per watt with some consideration for developer friendlyness, I'm pretty sure in 100 years we will still be coding minimalist JavaSE on the server and vanilla C (compiled with C++ compiler) on the client.
- Jodd – The Unbearable Lightness of Java
What are some alternatives?
spark-fast-tests - Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
spark-daria - Essential Spark extensions and helper methods ✨😲
huproxy
quinn - pyspark methods to enhance developer productivity 📣 👯 🎉
cmdg - Command line Gmail client
Nullboard - Nullboard is a minimalist kanban board, focused on compactness and readability.
lowdefy - The config web stack for business apps - build internal tools, client portals, web apps, admin panels, dashboards, web sites, and CRUD apps with YAML or JSON.
GoJS, a JavaScript Library for HTML Diagrams - JavaScript diagramming library for interactive flowcharts, org charts, design tools, planning tools, visual languages.
cakephp-swagger-bake - Automatically generate OpenAPI, Swagger, and Redoc documentation from your existing CakePHP code.
null - Nullable Go types that can be marshalled/unmarshalled to/from JSON.
Aerospike - Aerospike Database Server – flash-optimized, in-memory, nosql database
dbmate - :rocket: A lightweight, framework-agnostic database migration tool.