The pages dataset contains a new field in the JSON-encoded HAR payloads: _CrUX. As you can guess, this represents the real-user field data from the Chrome UX Report (CrUX) dataset. This presents a really interesting opportunity for web transparency research because previously the only queryable dataset with field data from the web at large was the public CrUX dataset on BigQuery. One major limitation of that dataset is the fact that it’s only available at the origin level. This new field in HTTP Archive is at the URL level, made possible by WebPageTest invoking the CrUX API for every URL at test time. So what can we do with URL-level CrUX data in HTTP Archive?
Here’s a query that joins the URL-level CrUX field data with the lab-based Lighthouse data to try to answer questions about the correlations between the Lighthouse performance score and the Core Web Vitals performance.
SELECT
COUNT(0) AS urls,
CORR(lh_perf_score, pct_good_lcp) AS lcp_correlation,
CORR(lh_perf_score, pct_good_fid) AS fid_correlation,
CORR(lh_perf_score, pct_good_cls) AS cls_correlation
FROM (
SELECT
url,
CAST(JSON_QUERY(report, "$.categories.performance.score") AS FLOAT64) AS lh_perf_score
FROM
`httparchive.lighthouse.2021_05_01_mobile`)
JOIN (
SELECT
url,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.largest_contentful_paint.histogram[0].density") AS FLOAT64) AS pct_good_lcp,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.first_input_delay.histogram[0].density") AS FLOAT64) AS pct_good_fid,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.cumulative_layout_shift.histogram[0].density") AS FLOAT64) AS pct_good_cls
FROM
`httparchive.pages.2021_05_01_mobile`)
USING
(url)
| urls | lcp_correlation | fid_correlation | cls_correlation |
|---|---|---|---|
| 7282377 | 0.49 | -0.14 | 0.37 |
Surprisingly, the correlations for each CWV are weak to medium strength and for FID it’s actually a negative correlation, meaning that as the Lighthouse score goes up, real-user FID performance actually tends to go down a bit. Of all the metrics, this makes the most sense for FID because it’s the hardest to emulate in the lab.
This uses the Lighthouse version 7.0 weighting for the performance score, which is based on TBT as a proxy for FID. The score weights TBT and LCP the highest at 25%, with CLS weighted at 5%. The v8.0 weighting changes these metrics to 30%, 25%, and 15% respectively and we’ll have the v8.0 results to analyze at the end of June 2021. It’ll be interesting to see how the correlations change across versions.
One way to visualize this is with a scatterplot. Here’s a query that takes a sample of 1000 representative data points representing the Lighthouse score and CWV performance for each page:
SELECT
* EXCEPT (i, n)
FROM (
SELECT
ROW_NUMBER() OVER (ORDER BY lh_perf_score) AS i,
COUNT(0) OVER () AS n,
lh_perf_score,
pct_good_cls,
pct_good_fid,
pct_good_lcp
FROM (
SELECT
url,
CAST(JSON_QUERY(report, "$.categories.performance.score") AS FLOAT64) AS lh_perf_score
FROM
`httparchive.lighthouse.2021_05_01_mobile`)
JOIN (
SELECT
url,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.largest_contentful_paint.histogram[0].density") AS FLOAT64) AS pct_good_lcp,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.first_input_delay.histogram[0].density") AS FLOAT64) AS pct_good_fid,
CAST(JSON_QUERY(payload, "$._CrUX.metrics.cumulative_layout_shift.histogram[0].density") AS FLOAT64) AS pct_good_cls
FROM
`httparchive.pages.2021_05_01_mobile`)
USING
(url)
WHERE
lh_perf_score IS NOT NULL AND
pct_good_cls IS NOT NULL AND
pct_good_fid IS NOT NULL AND
pct_good_lcp IS NOT NULL)
WHERE
MOD(i, CAST(FLOOR(n / 1000) AS INT64)) = 0
View the results in a spreadsheet
The built-in trend line feature in Google Sheets corroborates the Pearson coefficients calculated directly from BigQuery, with the FID trend going down and the LCP trend slightly steeper than that of CLS.
So to sum up, the Lighthouse performance score is most closely correlated with good LCP values, but not very strongly. It correlates even less strongly with CLS, and slightly negatively with FID.
