Post Mortem on Mastodon Outage with 30k users

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • Collectd

    The system statistics collection daemon. Please send Pull Requests here!

  • Then you will have same problems but now you can bother manufacturer about it!

    Also unless there is something horribly wrong about how often data is written, that SSD should run for ages.

    We ran (for a test) consumer SSDs in busy ES cluster and they still lasted like 2 years just fine

    The whole setup was a bit of overcomplicated too. RAID10 with 5+1 or 7+1 (yes Linux can do 7 drive RAID10) with hotspare woud've been entirely fine, easier, and most likely faster. You need backups anyway so ZFS doesn't give you much here, just extra CPU usage

    Either way, monitoring wait per drive (easy way is to just plug collectd [1] into your monitoring stack, it is light and can monitor A TON of different metrics)

    * [1]https://collectd.org/

  • syncthing-android

    Wrapper of syncthing for Android.

  • even some hacked setup with 2 way sync is better. The dumbest (but still somehow the least problem in their architecture) I saw was running [1] syncthing on few servers and just syncing everything and hoping conflicts won't happen...

    Like, I did it, but I wouldn't recommend it, restarting NFS server is gnarly, HA support on OSS side is... not really there last time I checked, and overall PITA.

    * [1] https://syncthing.net/

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • instances

    Mastodon instances list

  • Trollish usernames aren't allowed on HN because they subtly troll every thread they post to. I've therefore banned this account. If you don't want it to stay banned, we can rename it for you.

    https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts