Top 23 SQL Open-Source Projects
-
Project mention: Show HN: Box – Data Transformation Pipelines in Rust DataFusion | news.ycombinator.com | 2021-11-30
A while ago I posted a link to [Arc](https://news.ycombinator.com/item?id=26573930) a declarative method for defining repeatable data pipelines which execute against [Apache Spark](https://spark.apache.org/).
Today I would like to present a proof-of-concept implementation of the [Arc declarative ETL framework](https://arc.tripl.ai) against [Apache Datafusion](https://arrow.apache.org/datafusion/) which is an Ansi SQL (Postgres) execution engine based upon Apache Arrow and built with Rust.
The idea of providing a declarative 'configuration' language for defining data pipelines was planned from the beginning of the Arc project to allow changing execution engines without having to rewrite the base business logic (the part that is valuable to your business). Instead, by defining an abstraction layer, we can change the execution engine and run the same logic with different execution characteristics.
The benefit of the DataFusion over Apache Spark is a significant increase in speed and reduction in execution resource requirements. Even through a Docker-for-Mac inefficiency layer the same job completes in ~4 seconds with DataFusion vs ~24 seconds with Apache Spark (including JVM startup time). Without Docker-for-Mac layer end-to-end execution times of 0.5 second for the same example job (TPC-H) is possible. * the aim is not to start a benchmarking flamewar but to provide some indicative data *.
The purpose of this post is to gather feedback from the community whether you would use a tool like this, what features would be required for you to use it (MVP) or whether you would be interested in contributing to the project. I would also like to highlight the excellent work being done by the DataFusion/Arrow (and Apache) community for providing such amazing tools to us all as open source projects.
-
Project mention: Comparing Nginx Performance in Bare Metal and Virtual Environments | news.ycombinator.com | 2021-10-29
I do agree with you in that regard, however, that's also a dangerous line of thinking.
There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility
There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.
Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.
Alternatively, consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.
If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.
To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible.
-
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
Project mention: What is the consensus about using ORM in node js applications? | reddit.com/r/node | 2021-11-28
-
-
Recently a colleague, Nathan, reflecting on CockroachDB, remarked (paraphrased from memory) that the key data structure is the interval btree. The story of Nathan’s addition of the first interval btree to cockroach and the power of copy-on-write data structures is worthy of its own blog post for another day. It’s Nathan’s hand-specialization of that data structure that provided the basis (and tests) for the generalization I’ll be presenting here. The reason for this specialization was as much for the performance wins of avoiding excessive allocations, pointer chasing, and cost of type assertions when using interface boxing.
-
There's ksqldb (open source, built with java) and materialize (there's standalone edition), both need to use Kafka/RedPanda, also Clickhouse (open source, with materialize view with specific engine, but need to buffer the inserts using proxy like KittenHouse or buffering library like ch-timed-buffer), is there any other alternative to those 3 (that similarly doesn't do full scan to do aggregation)?
-
devops-exercises
Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions
Project mention: What language should I learn after Java and Python and how should I use my knowledge to learn about their applications in the real world? | reddit.com/r/computerscience | 2021-11-21What about the applications in the real world? Github are the answer. Some weeks ago I found a repository where a list of ideas to create your technologies show: link. If you are lost, try to guide yourself through roadmaps. Or just search for exercises on GitHub about your profession. Example: DevOps Exercises.
-
-
Project mention: Open Source Tanium Alternative (Cannot Remember It's Name) | reddit.com/r/sysadmin | 2021-11-22
You might be thinking of osquery?
-
How about batch insert, updates, and deletes? I had to fix a broken MyBatis project recently and was surprised that this feature doesn't even seem to be implemented, at least according to this GitHub issue.
-
Knex
A query builder for PostgreSQL, MySQL, CockroachDB, SQL Server, SQLite3 and Oracle, designed to be flexible, portable, and fun to use.
Knex.js works better than node-postgres at least in the cases where you need to dynamically build a SQL query based on (URL) arguments. For example:
-
Project mention: Updates and FAQ — Your 1 Minute Quick Start Guide to ShardingSphere | dev.to | 2021-11-30
shardingsphere-example is an independent Maven project. It’s preserved in the “examples” file of Apache ShardingSphere. [Link]：(https://github.com/apache/shardingsphere/tree/master/examples)
-
The "legacy stack" as it came to be known was written in .Net 4.5.3 using Entity Framework and the classic repository / unit of work pattern. As we worked with Entity Framework Core in .Net Core we found that we were not improving our query speeds. They were still slow. Usually in the area of 250ms on up based on the query. Entity Framework wasn't going to cut it for this "new stack" code. We decided to try Dapper. Along with Dapper we decided to adopt a different pattern with how our back end code would be structured and data would be delivered. After reading quite a bit about CQRS (Command Query Responsibility Segregation) and finding some great examples online we settled on this pattern. This article in particular was very useful. Though we didn't follow it exactly, we stole quite a few ideas from it.
-
Project mention: Let's write a compiler, part 5: A code generator | news.ycombinator.com | 2021-08-19
-
TimescaleDB
An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
Project mention: Entity Relationship Diagram for an Equities Database (finance/stocks) | reddit.com/r/Database | 2021-11-10you probably want a time series database like https://www.timescale.com/
-
go-sql-driver/mysql
Go MySQL Driver is a MySQL driver for Go's (golang) database/sql package (by go-sql-driver)
For DB connection, you'll need something like https://github.com/Go-SQL-Driver/MySQL/ for mysql
-
-
Project mention: DuckDB-WASM: Efficient Analytical SQL in the Browser | news.ycombinator.com | 2021-10-29
-
Project mention: Cloudflare Durable Objects Are Now Generally Available | news.ycombinator.com | 2021-11-15
-
Project mention: Compile Python applications into stand-alone executables | news.ycombinator.com | 2021-12-03
How did you go about developing against the Starlark API, any IDE support?
-
That's really bad security. Every password should be unique. Then, if someone gets one password, they don't have access to everything. If you have trouble remembering passwords, use a password manager like BitWarden
-
beekeeper-studio
Modern and easy to use SQL client for MySQL, Postgres, SQLite, SQL Server, and more. Linux, MacOS, and Windows.
Project mention: Any developer tools/subscriptions/packages that are worth buying paid/pro plan? | reddit.com/r/webdev | 2021-11-25In case anyone is looking for an alternative, Beekeeper Studio is decent (and free). It's not flawless, and it doesn't have as many features yet, but it's faster and less buggy on Windows at least.
-
A web application wont be complete without one, using a nifty tool called diesel we'll be able to create database tables, schemas and make queries in Rust but first we need a database installed, ill be using postgresql, also make sure you install just the postgresql feature of diesel.
-
Migrations: migrate
SQL related posts
- Made a Programing language using python
- questdb: NEW Data - star count:7279.0
- I would like to code in Rust. What can you suggest?
- questdb: NEW Data - star count:7279.0
- 1.3 GB CSV imported via EF Core in 6 min with EFCore.BulkExtensions
- How do I add a db to an existing connection other than the one it asks for when connecting? I tried refreshing and messing around with options but found nothing. (I want both root and simple_bank)
- questdb: NEW Data - star count:7279.0
Index
What are some of the best open-source SQL projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Apache Spark | 31,485 |
| 2 | tidb | 29,718 |
| 3 | Sequelize | 25,332 |
| 4 | dbeaver | 23,407 |
| 5 | cockroach | 22,567 |
| 6 | ClickHouse | 20,779 |
| 7 | devops-exercises | 19,638 |
| 8 | OSQuery | 18,424 |
| 9 | MyBatis | 16,556 |
| 10 | Knex | 14,990 |
| 11 | shardingsphere | 14,919 |
| 12 | Dapper | 14,268 |
| 13 | Presto | 12,902 |
| 14 | TimescaleDB | 12,140 |
| 15 | go-sql-driver/mysql | 11,690 |
| 16 | dolt | 9,800 |
| 17 | sql.js | 9,547 |
| 18 | rqlite | 9,085 |
| 19 | q | 8,765 |
| 20 | Bitwarden | 8,716 |
| 21 | beekeeper-studio | 7,860 |
| 22 | diesel | 7,632 |
| 23 | migrate | 7,585 |
Are you hiring? Post a new remote job listing for free.
