dupver
pgsink
Our great sponsors
dupver | pgsink | |
---|---|---|
8 | 5 | |
13 | 76 | |
- | - | |
0.0 | 0.0 | |
over 1 year ago | about 1 year ago | |
Go | Go | |
BSD 2-clause "Simplified" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dupver
-
Data Version Control
I work with a lot of uncompressed structured binary files so I finally broke down and wrote my own system based on the Restic chunker: https://github.com/akbarnes/dupver
-
Write Plain Text Files
I wound up writing dupver https://github.com/akbarnes/dupver after getting frustrated with the lack of versioning tools for binary files. One neat thing about .docx files and their ilk is that they are "just" zip files so it isn't hard to add special handling to pull out their contents and run deduplication over that.
- Dupver - Deduplicating VCS for large binary files in Go
- Show HN: Deduplicating VCS for large binary files in Go
- Dupver: deduplicating version control for large-ish binary files
-
Ask HN: Show me your Half Baked project
DupVer https://github.com/akbarnes/dupver is a deduplicating version control system for large binary files. It's designed to keep state in a repository on the local machine separate from the working directory so it plays nice with cloud synchronization software.
I started it after constant headaches involving Git LFS and the corporate proxy. It's based around the Restic chunker library, with inspiration from both the Duplicacy backup software and Boar, another binary version control system for large binary files.
-
What comes after Git? It's been 15 years since it was created
https://github.com/akbarnes/dupver
pgsink
-
GitHub - go-jet/jet: Type safe SQL builder with code generation and automatic query result data mapping
This is a really awesome project. I’ve used it on https://github.com/lawrencejones/pgsink to generate type safe bindings to the Postgres catalog tables, along with a few of the tables the project maintains itself.
-
Trade-offs from using ULIDs at incident.io
pgx is really good: it's what I used to write logical decoders in https://github.com/lawrencejones/pgsink
-
A modern data stack for startups
It used to be that companies would write their own hacky scripts to perform this extraction - I've had terrible incidents caused by ETL database triggers in the past, and even built a few generic ETL tools myself.
- Sync Postgres to BigQuery, possible? How?
-
Ask HN: Show me your Half Baked project
Postgres change-capture device that supports high-throughput and low-latency capture to a variety of sinks (at first release, just Google BigQuery):
https://github.com/lawrencejones/pgsink
I know there's debezium and Netflix's dblog, but this project aims to be much simpler.
Forget about kafka and any other dependency: just point it at Postgres, and your data will be pushed into BigQuery. And for people with highly-performance-sensitive databases, the read workload has been designed with Postgres efficiency in mind.
I'm hoping pgsink could be a gateway drug to get small companies up and running with a data warehouse. If your datastore of choice is Postgres, it's a huge help to replicate everything into an analytics datastore. A similar tool has helped my company extract expensive work out of our primary database, which is super useful for scaling.
The project is 90% there, about 10hrs and some testing away from being useable. Once there, I'll be hitting up some start-up friends and seeing if they want to give it a whirl.
What are some alternatives?
wcp
pastty - Copy and paste across devices
qrono - Qrono time-ordered queue server
DataflowTemplates - Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
UsTaxes - Tax filing web application
debezium-examples - Examples for running Debezium (Configuration, Docker Compose files etc.)
mymusic-dl - Download music using web scraping and youtube-dl no API keys required
xact - Model based design for developers
tinyjam - A radically simple, zero-configuration static site generator in JavaScript
dbt-metabase - dbt + Metabase integration
godbledger - Accounting Software with GRPC endpoints and SQL Backends
thgtoa - The Hitchhiker’s Guide to Online Anonymity