Casync – A Content-Addressable Data Synchronization Tool

• casync

  5 1,412 0.0 C

  Content-Addressable Data Synchronization Tool

  

  I was wondering how this gets any common chunks at all with the removed file boundaries. Turns out that chunks don't have a set size, just min/max/avg values, so unaligned streams may end up synchronizing. https://github.com/systemd/casync/blob/master/src/cachunker.... If I understood that correctly, that's pretty cool.

  But looking at the code I'm having strong "nope" feelings. First, because of lines like "q += m, n -= m;". Second, because of int/enum/semantic abuse: `compression_type` may be _CA_COMPRESSION_TYPE_INVALID which I hope is 0, `>= 0` as a known compression type, or `-EAGAIN` as an error. (from https://github.com/systemd/casync/blob/99559cd1d8cea69b30022... ) I'd bet that just throwing afl at the decompressor will find issues :(

  I do like the idea though.

• zstd

  29 20,919 9.3 C

  Zstandard - Fast real-time compression algorithm

  

  Really wish this was part of official zstd (https://github.com/facebook/zstd/issues/395#issuecomment-535...) and not a contrib / separate tool.

• SonarQube

  www.sonarqube.org

  sponsored

  Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  

• desync

  0 276 0.0 Go

  Alternative casync implementation

  Mentioning the more portable desync is obligatory: https://github.com/folbricht/desync

• zstd-seekable-format-go

  0 59 0.0 Go

  Seekable ZSTD compression format implemented in Golang.

• bitkeeper

  2 353 2.7 C

  This is the master copy of the BitKeeper source

  

• bazel-cache

  0 0 0.0 Starlark

  Minimal cloud oriented Bazel gRPC cache

  I did PoC experiments with compression, chunking, and IPFS here: https://github.com/SaveTheRbtz/bazel-cache

  If you need a mature compression implementation for bazel I would recommend using recent bazel versions w/ gRPC-based bazel-remote: https://github.com/buchgr/bazel-remote

  bazel nowadays supports end-to-end compression w/ `--experimental_remote_cache_compression`: https://github.com/bazelbuild/bazel/pull/14041

• bazel-remote

  0 515 0.0 Go

  A remote cache for Bazel

  I did PoC experiments with compression, chunking, and IPFS here: https://github.com/SaveTheRbtz/bazel-cache

  If you need a mature compression implementation for bazel I would recommend using recent bazel versions w/ gRPC-based bazel-remote: https://github.com/buchgr/bazel-remote

  bazel nowadays supports end-to-end compression w/ `--experimental_remote_cache_compression`: https://github.com/bazelbuild/bazel/pull/14041

• InfluxDB

  www.influxdata.com

  sponsored

  Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  

• Bazel

  47 21,376 9.9 Java

  a fast, scalable, multi-language and extensible build system

  

  I did PoC experiments with compression, chunking, and IPFS here: https://github.com/SaveTheRbtz/bazel-cache

  If you need a mature compression implementation for bazel I would recommend using recent bazel versions w/ gRPC-based bazel-remote: https://github.com/buchgr/bazel-remote

  bazel nowadays supports end-to-end compression w/ `--experimental_remote_cache_compression`: https://github.com/bazelbuild/bazel/pull/14041

Suggest a related project