Advice on 100gbps WAN?

This page summarizes the projects mentioned and recommended in the original post on /r/HPC
Data Migration Parallel Rsync Packing

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • fpart

    Sort files and pack them into partitions

    • My favorite free solution is fpsync/fpart from https://github.com/martymac/fpart -- basically that is a highly optimized filesystem crawler and indexer that can spit out balanced lists of files to transfer to a waiting army of parallel rsync workers. Tools are provided to manage the rsync fleet. Combining fpsync/fpart with an army of parallel rsync workers is a great design pattern especially for HPC as you can farm the rsync workers out to compute nodes

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Am I crazy to expect 100gbps across the pacific ocean?

    2 projects | /r/networking | 1 Jul 2022

  • How to back up 100TB NAS to USB HDDs??

    1 project | /r/DataHoarder | 21 May 2021

  • Dealing with race conditions in ruby scripts run concurrently

    1 project | /r/rails | 24 May 2023

  • Rsync extremely slow on two ZFS local pools

    2 projects | /r/zfs | 6 Dec 2022

  • Rust newbie developed a tool to convert csv to sql

    3 projects | /r/rust | 11 May 2022