chispa
lowdefy
Our great sponsors
chispa | lowdefy | |
---|---|---|
12 | 49 | |
508 | 2,551 | |
- | 1.3% | |
6.7 | 9.6 | |
3 days ago | about 22 hours ago | |
Python | JavaScript | |
MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
chispa
-
Testing spark applications
Unit and e2e tests using a combination of pytest and chispa (https://github.com/MrPowers/chispa). Custom library to create random test data that fits schema with optional hardcoded overrides for relevant fields to test business logic.
-
Spark open source community is awesome
here's a little README fix a user pushed to chispa
-
Invitation to collaborate on open source PySpark projects
chispa is a library of PySpark testing functions.
-
installing pyspark on my m1 mac, getting an env error
The other approach I've used is Poetry, see the chispa project as an example. Poetry is especially nice for projects that you'd like to publish to PyPi because those commands are built-in.
-
Spark: local dev environment
- All Spark transformations are tested with pytest + chispa (https://github.com/MrPowers/chispa)
-
Pyspark now provides a native Pandas API
Pandas syntax is far inferior to regular PySpark in my opinion. Goes to show how much data analysts value a syntax that they're already familiar with. Pandas syntax makes it harder to reason about queries, abstract DataFrame transformations, etc. I've authored some popular PySpark libraries like quinn and chispa and am not excited to add Pandas syntax support, haha.
-
Show dataengineering: beavis, a library for unit testing Pandas/Dask code
I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.
-
Tips for building popular open source data engineering projects
Blogging has been the main way I've been able to attract users. Someone searches "testing PySpark", they see this blog, and then they're motivated to try chispa.
-
Ask HN: What are some tools / libraries you built yourself?
I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.
quinn (https://github.com/MrPowers/quinn) and chispa (https://github.com/MrPowers/chispa) are the PySpark equivalents.
Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.
Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8
-
Open source contributions for a Data Engineer?
I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.
lowdefy
-
Pkl, a Programming Language for Configuration
I'm really enjoying reading through the docs and the tutorial. We've created Lowdefy, a config web-stack which makes it really simple to build quite advanced web apps. We're writing everything in YAML, but it has it's limitations, specifically when doing config type checking and IDE extensions that go beyond just YAML.
I've been looking for a way to have typed objects in the config to do config suggestions and type checking.. PKL looks like it can do this for us. And with the JSON output we might even be able to get there with minimal effort.
Is there anyone here with some PKL experience that would be willing to answer some technical questions re the use of PKL for more advanced, nested config?
See Lowdefy:
https://lowdefy.com/
https://github.com/lowdefy/lowdefy
-
Show HN: Retool AI
Awsome! With Lowdefy we tried to build a low-code framework that works like code. We’ve developed a schema in which to define applications and we’ve built all kinds of apps for enterprise customers. Massive, advanced CRM systems, call centre solutions, ticketing systems, a light MRP, all kinds of survey apps and so many dashboards. Even our docs and our website are Lowdefy apps!
Give Lowdefy a try and reach out it you have any questions or want to see what is possible :) (We need to invest a lot more into content and examples, bootstapping is a grind!)
https://github.com/lowdefy/lowdefy
-
Launch HN: Refine (YC S23) – Open-Source Retool for Enterprise
Also add Lowdefy onto the list https://github.com/lowdefy/lowdefy
co-founder here :)
-
The Surprising Power of Documentation
100% this. And yes, good documentation takes a lot of investment but it pays off like compound interest. But with that done, it becomes even more important not to pull the carpet for no good reason, you are building a tower and documentation is at the foundation.
We’ve built Lowdefy [1] as an open source project and documented it with all effort, 200 pages of docs. I often forget why or how something works and then jump to the docs. This investment keeps on paying of as we use Lowdefy to build customer apps, new devs in the team typically take less than two week to get up to speed and start making contributions, the sharp ones, just a two or three days.
This year, we’re extended our documentation onto customer apps aswell, with flow diagrams, state machine definitions, detailed field level explication schema definitions, and end user test procedures. The key here for this documentation is detail. It should be easier to reach for the docs and the the answer, than to dive in the code and interpret it.
1 - https://github.com/lowdefy/lowdefy
-
how to choose a tech stack for a personal project
https://github.com/lowdefy/lowdefy Co-Founder here.
-
Ask HN: What have you built more than twice and wish someone had built for you?
Check out https://lowdefy.com/ they even have a sample survey app as one of their examples.
-
Looking for a workflow program, any suggestions?
You can build an app that would do this
-
AG Grid Community Roundup July 2022
Lowdefy is a low code tool that uses AG Grid as a block component, allowing you to create apps which render data in AG Grid without a lot of coding knowledge. There is a Lowdefy example using AG Grid here.
-
Story of raising VC funding for my open-source project
Shameless plug, also check out Lowdefy - https://github.com/lowdefy/lowdefy
-
Show HN: ToolJet 1.2 OSS Retool alternative with realtime multiplayer editing
I’m also going to jump in here and say try Lowdefy https://github.com/lowdefy/lowdefy - co-founder here.
We take a different angle and believe that low code should still work like code. We focus on a developer first approach.
What are some alternatives?
spark-fast-tests - Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
appsmith - Platform to build admin panels, internal tools, and dashboards. Integrates with 25+ databases and any API.
spark-daria - Essential Spark extensions and helper methods ✨😲
budibase - Budibase is an open-source low code platform that helps you build internal tools in minutes 🚀
quinn - pyspark methods to enhance developer productivity 📣 👯 🎉
ToolJet - Low-code platform for building business applications. Connect to databases, cloud storages, GraphQL, API endpoints, Airtable, Google sheets, OpenAI, etc and build apps using drag and drop application builder. Built using JavaScript/TypeScript. 🚀
null - Nullable Go types that can be marshalled/unmarshalled to/from JSON.
streamlit - Streamlit — A faster way to build and share data apps.
dagster - An orchestration platform for the development, production, and observation of data assets.
QR-Code-generator - High-quality QR Code generator library in Java, TypeScript/JavaScript, Python, Rust, C++, C.
fugue - A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
authentik - The authentication glue you need.