Cascade of doom: JIT, and how a Postgres update led to 70% failure on a critical national service

• coronavirus-dashboard-summary

  3 54 6.2 F#

  UK Coronavirus Dashboard - Summary pages

  

We released an new version of summary pages implemented in F#.

• coronavirus-dashboard-pipeline-etl

  1 5 7.2 Python

  UK Coronavirus Dashboard ETL

  

use the cache pre-population solution implemented in our ETL

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• pgbouncer

  34 2,648 8.7 C

  lightweight connection pooler for PostgreSQL

  

Towards the end of the week, a few people reached out to me saying that they were encountering an increasing number of failed requests when using APIv2. They were peculiarly saying that they couldn't download new data from the API after the release. My first action in such cases is to check the API service, and our PGBouncer instances to ensure that they are healthy. Then I usually clear the storage cache for APIv2, which usually solves the problem.

• coronavirus-dashboard-generic-apis

  1 6 3.0 Go

  Coronavirus Dashboard (COVID-19) in the UK - Generic APIs

  

Number of grievances directed to me in Twitter increased. People were now reporting increased latency in our Generic API. This concerned me because unlike APIv2, which is aggressively throttled and is designed to download very large payloads, the Generic API has been implemented in the Go programming language using highly optimised queries for rapid response. Whilst a download from APIv2 may take 3 or 4 minutes to complete, the average latency for our Generic API is ~50 milliseconds.

• llvm-project

  349 25,563 10.0 C++

  The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.

  

Based on discussions in the issue ticket, it appears that the problem is associated with a setting entitled Just-in-time Compilation (JIT) that is turned on by default in Postgres 14 when it is compiled using LLVM.

• asyncpg

  15 6,609 7.2 Python

  A fast PostgreSQL Database Client Library for Python/asyncio.

  

Simple query runs long when DB schema contains thousands of tables #186

• coronavirus-dashboard

  800 251 7.1 JavaScript

  Dashboard for tracking Coronavirus (COVID-19) across the UK

  

The UK coronavirus dashboard is the primary data reporting service for the COVID-19 pandemic in the United Kingdom. It sustains on average 45 to 50 million hits a day, and is regarded as a critical service.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project