quinn
rupy
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
quinn
-
PySpark OSS Contribution Opportunity
Adding some README documentation to the README should be quite straightforward. Here's a function that needs to be documented: https://github.com/MrPowers/quinn/issues/52 .
There are a lot of issues in the quinn repo with a "good first issue" tag if you'd like to get started: https://github.com/MrPowers/quinn/issues
-
Invitation to collaborate on open source PySpark projects
quinn is a library with PySpark helper functions. I need to work through all the open issues / PRs and bump all versions. I should do another release. This library gets around 600,000 monthly downloads.
-
Pyspark now provides a native Pandas API
Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.
-
Is Spark - The Defenitive Guide outdated?
They spent a lot of effort improving the catalyst engine under the hood too and making it easier to extend and improve it in the future. Making it easy to add your own native code to Spark itself. Shameless plug of a blog post I wrote on this subject which basically reiterates what Matthew Powers, author of Spark Daria and quinn, wrote here.
-
Ask HN: What are some tools / libraries you built yourself?
I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.
quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.
Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.
Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8
-
Open source contributions for a Data Engineer?
I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.
rupy
-
Considerations for a long-running Raspberry Pi
I have been running a Raspberry 2 cluster for 10 years: http://host.rupy.se
A few weeks back the first SD card to fail got so corrupted it failed to reboot!
My key learning is use oversized cards, because then the bitcycle will wear slower!
I'm going from 32GB to 256/512/1024!
- Sandstorm: Open-source platform for self-hosting web app
-
You Want Modules, Not Microservices
I think we're all confused over the definition. Also one might understand what all the proponents are talking about better if they think about this more as a process and not some technological solution:
https://github.com/tinspin/rupy/wiki/Process
All input I have is you want your code to run on many machines, in fact you want it to run the same on all machines you need to deliver and preferably more. Vertically and horizontally at the same time, so your services only call localhost but in many separate places.
This in turn mandates a distributed database. And later you discover it has to be capable of async-to-async = no blocking ever anywhere in the whole solution.
The way I do this is I hot-deploy my applications async. to all servers in the cluster, this is what a cluster node looks like in practice (the name next to Host: is the node): http://host.rupy.se if you click "api & metrics" you'll see the services.
With this not only do you get scalability, but also redundancy and development is maintained at live coding levels.
-
I wish my web server were in the corner of my room
I have hosted my own web server both physically and codevise since 2014.
It's on a Raspberry 2 cluster:
Since 2016 i have my own database also coded from scratch:
We need to implement HTTP/1.1 with less bloat, a C non-blocking web server that can share memory between threads is probably the most interesting project for humans right now, is anyone working on that?
-
Ask HN: Free and open source distributed database written in C++ or C
I have one in Java: https://github.com/tinspin/rupy
Here is the 2000 lines of code of the entire database: http://root.rupy.se/code?path=/Root.java
And here you can try it out: http://root.rupy.se
-
Dokku – Free Heroku Alternative
The smallest PaaS you have ever seen is one order of magnitude larger than mine: https://github.com/tinspin/rupy
And I bet you the same goes for performance, if not two!
-
Server-Sent Events: the alternative to WebSockets you should be using
Absolutely not, HTTP/1.1 is the way to make SSE fly:
https://github.com/tinspin/rupy/wiki/Comet-Stream
Old page search for "event-stream"... Comet-stream is a collection of techniques of which SSE is one. My findings are that SSE go through anti-viruses better!
I would look at my own app-server: https://github.com/tinspin/rupy
It's not the most well documented but it's the smallest implementation while still being one of the most performant so you can learn more than just SSE.
The data is here: http://fuse.rupy.se/about.html
Under Performance. Per watt the fuse/rupy platform completely crushes all competition because of 2 reasons:
- Event driven protocol design, averages at about 4 messages/player/second (means you cannot do spraying or headshots f.ex. which is another feature in my game design opinion).
- Java's memory model with atomic concurrency which needs a VM and GC (C++ copied that memory model in C++11, but it failed completely because they lack both VM and GC, but that model is still to this day the one C++ uses), you can read more about this here: https://github.com/tinspin/rupy/wiki
You can argue those points are bad arguments, but if you look at performance per watt with some consideration for developer friendlyness, I'm pretty sure in 100 years we will still be coding minimalist JavaSE on the server and vanilla C (compiled with C++ compiler) on the client.
- Jodd – The Unbearable Lightness of Java
What are some alternatives?
huproxy
chispa - PySpark test helper methods with beautiful error messages
spark-daria - Essential Spark extensions and helper methods ✨😲
cmdg - Command line Gmail client
Nullboard - Nullboard is a minimalist kanban board, focused on compactness and readability.
spark-rapids - Spark RAPIDS plugin - accelerate Apache Spark with GPUs
dbmate - :rocket: A lightweight, framework-agnostic database migration tool.
GoJS, a JavaScript Library for HTML Diagrams - JavaScript diagramming library for interactive flowcharts, org charts, design tools, planning tools, visual languages.
cakephp-swagger-bake - Automatically generate OpenAPI, Swagger, and Redoc documentation from your existing CakePHP code.
Aerospike - Aerospike Database Server – flash-optimized, in-memory, nosql database
simonw - https://simonwillison.net/2020/Jul/10/self-updating-profile-readme/
Pion WebRTC - Pure Go implementation of the WebRTC API