Why Is Jepsen Written in Clojure?

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • kit

    Lightweight, modular framework for scalable web development in Clojure (by kit-clj)

  • I am not sure what a web framework is, to be honest. The choices for many parts of a web application are really domain-specific and I'm not sure a single "framework" would work for everyone.

    As far as web-related components go, my app uses Rum (as an interface to React), ring, http-kit, pushy (for history manipulation), sente (for websockets), buddy (for authentication tools).

    If you are looking for a batteries-included "I want to have some sort of webapp right away" thing, I think https://kit-clj.github.io would fit the bill, but the general feeling in the Clojure community is that unlike Python with Django or Ruby with Rails, the choice of app components is not predetermined by the language.

  • biff

    A Clojure web framework for solo developers.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
  • Korma

    Tasty SQL for Clojure.

  • elle

    Black-box transactional safety checker based on cycle detection

  • Speaking very loosely, primitives on the JVM are values which are represented directly in memory, instead of as pointers to objects on the heap. Clojure generally treats everything as a pointer to a heap object. There is no specialized equivalent for, say, a vector of shorts, or a map where values are floats. The compiler can emit specialized function signatures for... IIRC longs and doubles, but other types (e.g. byte, float) aren't directly accessible--they go through widening conversion. It's also easy for the compiler to quietly fail to recognize it can preserve primitives in some kinds of loops, so you wind up with what Java calls "autoboxing": wrapping a primitive in a corresponding Object type.

    Here's a recent example of some code in a hot path inside Elle, one of Jepsen's safety checkers. It does a lot in primitive, packed structs and bitmasks to avoid pointer chasing.

    https://github.com/jepsen-io/elle/blob/main/src/elle/BFSPath...

    There was actually a Clojure version of this earlier that got pretty close perf-wise, but I wound up dropping to Java for it instead:

    https://github.com/jepsen-io/elle/blob/913cbff5ebb19ba850c0a...

  • dom-top

    Unorthodox control flow, for Clojurists with masochistic sensibilities.

  • Nope! For a few reasons Jepsen sticks very closely to Traditional JVM Threads.

    Part of that is because Jepsen isn't hyper-concurrent. Most concurrency errors manifest in just a handful of client threads, and the JVM will run ~10K threads without too many issues.

    Another part is that I spend a lot of time debugging and profiling. Core.async turns stacktraces into Swiss cheese--it has to, to perform its implicit continuation-passing magic. But that can make it really hard to figure out Who Called What When, and Why Is This Process Slow. Standard blocking IO calls are also something that my profiler (YourKit) can analyze well.

    Finally, I invest a lot of time into tuning Jepsen for speed, and manage the handoff of data between threads carefully. In a system where actors were handing off state constantly core.async would be a bigger help, but there's really only a couple shapes for concurrent handoff in Jepsen, and they're addressed by either j.u.c. queues, forkjoin-style structured concurrency, or Weird Transactional Threadpool stuff.

    For a little more on this, take a look at https://github.com/aphyr/dom-top or https://github.com/jepsen-io/history/blob/main/src/jepsen/hi...

  • history

    Support library for working with Jepsen histories (by jepsen-io)

  • Nope! For a few reasons Jepsen sticks very closely to Traditional JVM Threads.

    Part of that is because Jepsen isn't hyper-concurrent. Most concurrency errors manifest in just a handful of client threads, and the JVM will run ~10K threads without too many issues.

    Another part is that I spend a lot of time debugging and profiling. Core.async turns stacktraces into Swiss cheese--it has to, to perform its implicit continuation-passing magic. But that can make it really hard to figure out Who Called What When, and Why Is This Process Slow. Standard blocking IO calls are also something that my profiler (YourKit) can analyze well.

    Finally, I invest a lot of time into tuning Jepsen for speed, and manage the handoff of data between threads carefully. In a system where actors were handing off state constantly core.async would be a bigger help, but there's really only a couple shapes for concurrent handoff in Jepsen, and they're addressed by either j.u.c. queues, forkjoin-style structured concurrency, or Weird Transactional Threadpool stuff.

    For a little more on this, take a look at https://github.com/aphyr/dom-top or https://github.com/jepsen-io/history/blob/main/src/jepsen/hi...

  • clojure-graph-resources

    A curated list of Clojure resources for dealing with graph-like data.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • honeysql

    Turn Clojure data structures into SQL

  • I recall using korma way back I and I don’t recall it being terrible but I would say https://github.com/seancorfield/honeysql has very much superseded it by this point… (but I can see how that might not be obviously clear if one is to look at superficial metrics like GitHub stars for example…)

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts