almanac.httparchive.org VS httparchive.org

The CrUX dataset on BigQuery is aggregated at the origin level. An origin is a way to identify an entire website. For example, https://httparchive.org is the origin for the HTTP Archive website and it's different from https://almanac.httparchive.org, which is a separate origin for the Web Almanac website.

This is a community effort and we're a friendly, inclusive group, so if you'd like to be a part of the 2019-2020 translation effort or help write the 2021 edition, we'd love for you to join us! Please fill out this interest form to let us know how you'd like to contribute, and we'll reach out to you when it's time to start planning in mid-2021.

Astro didn't capture the data themselves, it's data from The HTTP Archive.

Based on HTTP Archive real-world performance data (Lighthouse, P90) • Read the full report

A longstanding problem in the HTTP Archive dataset has been extracting insights from blobs of HTML in BigQuery. For example, take the source code of example.com:

The data for this chart came from the wonderful httparchive.org. Tools used to make the chart: Python, Pandas, Matplotlib.

First of all, web development hasn't really evolved that much in the last decade, or it all depends on what you mean by “evolved”. Sure, there’s webpack and React and ES6, but according to data from httparchive.org the median page load time has remained about the same over the past 10 years, even though internet speeds have been steadily increasing, along with rapid advancements in hardware. So it’s safe to say that the internet is faster, but websites aren’t. In addition, as the team behind the Skypack utility has pointed out, “building for the web has never been more complicated.”

The technologies you use to build your website can have an effect on your ability to deliver good user experiences. Good UX is key to performing well with Core Web Vitals (CWV), a topic which is probably top of mind for you, as it is for many other web developers now that these metrics play a role in Google Search ranking. While web developers have had tools like Search Console and PageSpeed Insights to get data on how their sites are performing, the web community has been lacking a tool that has operated at the macro level, giving us something more like WebSpeed Insights. By combining the powers of real-user experiences in the Chrome UX Report (CrUX) dataset with web technology detections in HTTP Archive, we can get a glimpse into how architectural decisions like choices of CMS platform or JavaScript framework play a role in sites' CWV performance. The merger of these datasets is a dashboard called the Core Web Vitals Technology Report.

One of the main things that affect performance and is not so easy to optimize in a web application is assets such as images and videos. Images form almost half of all web content, and so optimizing them goes a very long way to improve experiences with applications. Images are unique because they are not usually compressed in an optimal way and they load all at once on the web and so the Next team decided to solve this. An image component in Next abstracts and enables images to be optimized without any input from you, the developer. Great right? All you have to do is replace the img element with Image from Next. Here is how it is done. Let’s say your image tag is like this:

The HTTP Archive runs Lighthouse audits for approximately 7.2 million websites every month. In the May 2021 dataset, Lighthouse was able to identify an LCP element in 97.35% of the tests. Since we have the ability to query all of these Lighthouse test results, we can analyze the result of the LCP audits and get more insight into what drives this metric across the web.

A good place for data around this is the HTTP Archive (https://httparchive.org/). It has collected data and reports.