peerdb
mini.nvim
peerdb | mini.nvim | |
---|---|---|
7 | 146 | |
1,816 | 3,945 | |
15.4% | - | |
9.9 | 9.6 | |
6 days ago | 8 days ago | |
Go | Lua | |
GNU General Public License v3.0 or later | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
peerdb
- PeerDB Streams – Simple, Native Postgres Change Data Capture
-
Pgwire: a Rust library for PostgreSQL compatible application
We at PeerDB (https://github.com/PeerDB-io/peerdb) were early adopters of Pgwire to implement our Postgres-compatible SQL Layer to do ETL. Very easy to work with. Saved us multiple months of effort to build it from scratch.
- FLaNK AI Weekly 18 March 2024
-
Show HN: Open-source x64 and Arm GitHub runners. Reduces GitHub Actions bill 10x
We've been using the Ubicloud runner for a while at PeerDB[1]. Great value and specially the ARM runners have been helpful to get our CI costs down. The team is really responsive and added the arm runner support within a few weeks of us requesting it.
[1] https://github.com/PeerDB-io/peerdb
-
Benchmarking Postgres Replication: PeerDB vs. Airbyte
Thanks for posting this question. Composite primary key support is actively being worked on and should be available in 1-2 weeks :) - https://github.com/PeerDB-io/peerdb/pull/499
-
Launch HN: PeerDB (YC S23) – Fast, Native ETL/ELT for Postgres
Hi HN! I'm Sai, the co-founder and CEO of PeerDB (https://www.peerdb.io/), a Postgres-first data-movement platform that makes moving data in and out of Postgres fast and simple. PeerDB is free and open (https://github.com/PeerDB-io/peerdb) and we provide a Docker stack for users to try us out. Our repo is at https://github.com/PeerDB-io/peerdb and there’s a 5-minute quickstart here: https://docs.peerdb.io/quickstart.
For the past 8 years, working at Microsoft on Postgres on Azure, and before that at Citus Data, I’ve worked closely with customers running Postgres at the heart of their data stack, storing anywhere from 10s of GB of data to 10s of TB.
This was when I got exposed to the challenges customers faced when moving data in and out of Postgres. Usually they would try existing ETL tools, fail, and decide to build in-house solutions. Common issues with these tools included painfully slow syncs - syncing 100s of GB of data took days; flaky and unreliable - frequent crashes, loss of data precision on target etc., and; feature-limited - lack of configurability, unsupported data types and so on.
I remember a specific scenario where a tool didn’t support something as simple as the Postgres’ COPY command to ingest data. This would have improved the throughput by orders of magnitude. We (customer and me) reached out to that company to request them to add this feature. They couldn’t prioritize this feature because it wasn’t very easy - their tech stack was designed to support 100s of connectors rather than supporting a native Postgres feature.
After multiple such occurrences, I thought, why not build a tool specialized for Postgres, making the lives of many Postgres users easier. I reached out to my long-time buddy Kaushik, who was building operating systems at Google and had led data teams at Safegraph and Palantir. We spent a few weeks building an MVP that streamed data in real-time from Postgres to BigQuery. It was 10 times faster than existing tools and maintained data freshness of less than 30 seconds. We realized that there were many Postgres native and infrastructural optimizations we could do to provide a rich data-movement experience for Postgres users. This is when we decided to start PeerDB!
We started with two main use cases: Real-time Change Data Capture from Postgres (demo: https://docs.peerdb.io/usecases/realtime-cdc#demo) and Real-time Streaming of query results from Postgres (demo: https://docs.peerdb.io/usecases/realtime-streaming-of-query-...). The 2nd demo shows PeerDB streaming a table with 100M rows from Postgres to Snowflake.
We implement multiple optimizations to provide a fast, reliable, feature-rich experience. For performance, we can parallelize the initial load of a large table, still ensuring consistency. Syncing 100s of GB goes from days to minutes. We do this by logically partitioning the table based on internal tuple identifiers (CTID) and parallelly streaming those partitions (inspired by this DuckDB blog - https://duckdb.org/2022/09/30/postgres-scanner.html#parallel...)
For CDC, we don’t use Debezium, rather handle replication more natively—reading the slot, replicating the changes, keeping state etc. We made this choice mainly for flexibility. Staying native helps us use existing and future Postgres enhancements more effectively. For example, if the order of rows across tables on the target is not important, we can parallelize reading of a single slot across multiple tables and improve performance. Our architecture is designed for real-time syncs, which enables data-freshness of a few 10s of seconds even at large throughputs (10k+ tps).
We have fault tolerance mechanisms for reliability (https://blog.peerdb.io/using-temporal-to-scale-data-synchron...) and support multiple features including log-based (CDC) / query based streaming, efficient syncing of tables with large (TOAST) columns, configurable batching and parallelism to prevent OOMs and crashes etc.
For usability - we provide a Postgres compatible SQL layer for data-movement. This makes the life of data engineers much easier. They can develop pipelines using a framework they are familiar with, without needing to deal with custom UIs and REST APIs. They can use Postgres' 100s of integrations to build and manage ETL. We extend Postgres' SQL grammar with a few new intuitive SQL commands to enable real-time data streaming across stores. Because of this, we were able to add dbt integration via Dagster (in private preview) in a few hours! We expect data-engineers to unravel similar integrations with PeerDB easily, and plan to make this grammar richer as we evolve.
PeerDB consists of the following components to handle data replication: (1) PeerDB Server uses the pgwire protocol to mimic a PostgreSQL server, responsible for query routing and generating gRPC requests to the Flow API. It relies on AST analysis to make informed decisions on routing. (2) Flow API: an API layer that deals with gRPC commands, orchestrating the data sync operations; (3) Flow Workers execute the data read-write operations from the source to the destination. Built to scale horizontally, they interact with Temporal for increased resilience. The types of data replication supported include CDC streaming replication and query-based batch replication. Workers do all of the heavy lifting, and have data store specific optimizations.
Currently we support 6 target data stores (BigQuery, Snowflake, Postgres, S3, Kafka etc) for data movement from Postgres. This doc captures the current status of the connectors: https://docs.peerdb.io/sql/commands/supported-connectors.
As we spoke to more customers, we realized that getting data into PostgreSQL at scale is equally important and hard. For example one of our customers wants to periodically sync data in multiple SQL Server instances (running on the edge) to their centralized Postgres database. Requests for Oracle to Postgres migrations are also common. So now we’re also supporting source data stores with Postgres as the target (currently SQL Server and Postgres itself, with more to come).
We are actively working with customers to onboard them to our self-hosted enterprise offering. Our fully hosted offering on the cloud is in private preview. We haven’t yet decided on the pricing. One common concern we’ve heard from customers is that existing tools are expensive and charge based on the amount of data transferred. To address this, we are considering a more transparent way of pricing—for example, pricing based on provisioned hardware (cpu, memory, disk). We’re open for feedback on this!
Check out our github repo - https://github.com/PeerDB-io/peerdb and go ahead and give it a spin (5-minute quickstart https://docs.peerdb.io/quickstart).
We want to provide the world’s best data-movement experience for Postgres. We would love to get your feedback on product experience, our thesis and anything else that comes to your mind. It would be super useful for us. Thank you!
mini.nvim
- FLaNK AI Weekly 18 March 2024
-
Alternative to vim-textmanip plugin? (move selected blocks of text)
This is essentially a tagline of mini.move.
-
Refactor files and update import paths
Just as the others suggested, oil.nvim solves this outta the box. I freaking love it (here my config in case ya need it). Apparently also mini.files handles this by default
-
Backwards inner/outer motions?
You mean backwards seeking text objects? You can get those with mini.ai https://github.com/echasnovski/mini.nvim/blob/main/readmes/mini-ai.md
-
mini.nvim - release 0.10.0 (files, clue, operators, and minor updates)
I would like to offer you to join me in saying late greetings to this autumn with a release of mini.nvim version 0.10.0. It is mostly about introducing three (quite feature full, dare I say) modules and minor updates of existing ones.
-
Shoutouts to u/echasnovski
Last night I cleaned up all the dead code in my config and realised that mini.nvim has to be the single best plugin that I've used. I have a couple of other favourites but this collection has been so consistently good that I wanted to give some thanks to the juggernaut that is u/echasnovski! Thanks for all the work you plugin authors and core maintainers put in to make this editor what it is <3
-
Looking for good tutorials for learning to use neovim as an IDE.
For example: I spent a lot of time configuring file tree plugins to have the same sorting as VS Code, tweaking their icons, etc. But then I realized I barely used the file explorer at all, and now I'm super happy with the minimal approach of mini.files. I had similar experiences with other plugins that were just adding "fluff" instead of the functionality I was looking for.
- F/f/T/t highlight plugin?
-
mini.files updates - file preview, prefix customization, change target window, and more
Around two weeks ago I've announced the release of mini.files - a file explorer module of mini.nvim with column view navigation and "edit text to manipulate file system" design. This resulted into a great feedback from the community, much of which turned into new features.
-
New plugin: Notebook Navigator - Execute and manipulate code cells a la VSCode
A mini.ai textobject specification that you can use standalone
What are some alternatives?
pglogical - Logical Replication extension for PostgreSQL 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
surround.nvim - A surround text object plugin for neovim written in lua. (Fork from blackCauldron7/surround.nvim)
transfer - Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
peek.nvim - Markdown preview plugin for Neovim
realtime - Broadcast, Presence, and Postgres Changes via WebSockets
specs.nvim - 👓 A fast and lightweight Neovim lua plugin to keep an eye on where your cursor has jumped.
cloudquery - The open source high performance ELT framework powered by Apache Arrow
leap.nvim - Neovim's answer to the mouse 🦘
materialize - The data warehouse for operational workloads.
lsp_lines.nvim - Mirror of https://git.sr.ht/~whynothugo/lsp_lines.nvim
bytebase - The GitHub/GitLab for database DevOps. World's most advanced database DevOps and CI/CD for Developer, DBA and Platform Engineering teams.
persistence.nvim - 💾 Simple session management for Neovim