# Analysis **URL:** https://discuss.httparchive.org/c/analysis/1.md?page=1 [Latest](https://discuss.httparchive.org/latest.md) · [Categories](https://discuss.httparchive.org/categories.md) **Page:** 2 --- ## [Querying the HTTP Archive with DuckDB](https://discuss.httparchive.org/t/querying-the-http-archive-with-duckdb/2568)
**Author:** [@acme](https://discuss.httparchive.org/u/acme)\ **Replies:** 0\ **Last updated:** [May 3, 2023, 10:51am UTC](https://discuss.httparchive.org/t/querying-the-http-archive-with-duckdb/2568 "2023-05-03T10:51:04Z")
I needed to explore ETag response headers locally and I’ve come up with a workflow for querying HTTP Archive data on my laptop in Parquet format using DuckDB. You may have to create Google Cloud project IDs, Google BigQ… --- ## [Why videos is not appearing on Median page weight by content type figure?](https://discuss.httparchive.org/t/why-videos-is-not-appearing-on-median-page-weight-by-content-type-figure/2565)
**Author:** [@bbarneoud](https://discuss.httparchive.org/u/bbarneoud)\ **Replies:** 3\ **Last updated:** [April 18, 2023, 11:57am UTC](https://discuss.httparchive.org/t/why-videos-is-not-appearing-on-median-page-weight-by-content-type-figure/2565 "2023-04-18T11:57:38Z")
Is it because videos are mainly tierce assets? How to compare videos weight regarding other content type in that case? --- ## [Interesting outliers](https://discuss.httparchive.org/t/interesting-outliers/2250)
**Author:** [@briankardell](https://discuss.httparchive.org/u/briankardell)\ **Replies:** 0\ **Last updated:** [November 2, 2021, 4:45pm UTC](https://discuss.httparchive.org/t/interesting-outliers/2250 "2021-11-02T16:45:32Z")
When I wrote the markup chapter for the almanac in 2019 we found a number of outliers - data that was sort of very “outside expectations”, and often same errors repeated across several sites. As we had some trouble gett… --- ## [Historical decline in WWW subdomain use?](https://discuss.httparchive.org/t/historical-decline-in-www-subdomain-use/2507)
**Author:** [@gwern](https://discuss.httparchive.org/u/gwern)\ **Replies:** 6\ **Last updated:** [February 9, 2023, 8:20pm UTC](https://discuss.httparchive.org/t/historical-decline-in-www-subdomain-use/2507 "2023-02-09T20:20:38Z")
It’s commonly said that the classic ‘www.’ subdomain is in steep decline, prompted by general lack of need and catering to mobile users. I agree that it does seem like you see more ‘foo.com’ rather than ‘www.foo.com’ eac… --- ## [Help with BigQuery to extract all datapoints for top 1,000 sites](https://discuss.httparchive.org/t/help-with-bigquery-to-extract-all-datapoints-for-top-1-000-sites/2544)
**Author:** [@matt](https://discuss.httparchive.org/u/matt)\ **Replies:** 1\ **Last updated:** [March 14, 2023, 9:36pm UTC](https://discuss.httparchive.org/t/help-with-bigquery-to-extract-all-datapoints-for-top-1-000-sites/2544 "2023-03-14T21:36:08Z")
Hi, would anyone be interested in helping me to extract the full data points for the top 1,000 domains on HTTP Archive for page size, and the worst 1,000 for size? Happy to pay for this, so please provide a rough quote. … --- ## [Cookie paywall](https://discuss.httparchive.org/t/cookie-paywall/2484)
**Author:** [@mbatikas](https://discuss.httparchive.org/u/mbatikas)\ **Replies:** 0\ **Last updated:** [December 1, 2022, 10:53am UTC](https://discuss.httparchive.org/t/cookie-paywall/2484 "2022-12-01T10:53:37Z")
Dear all, in Europe lately some news sites have adopted the practice of cookie paywall (Users have either to accept all cookies/trackers or pay in order to access the content of the news site). For more information you c… --- ## [What compression ratio ranges are used for lossy formats?](https://discuss.httparchive.org/t/what-compression-ratio-ranges-are-used-for-lossy-formats/2464)
**Author:** [@eeeps](https://discuss.httparchive.org/u/eeeps)\ **Replies:** 0\ **Last updated:** [November 2, 2022, 12:18am UTC](https://discuss.httparchive.org/t/what-compression-ratio-ranges-are-used-for-lossy-formats/2464 "2022-11-02T00:18:46Z")
In this year’s Web Almanac Media chapter, we published per-format median bit-per-pixel numbers, but not per-format distributions. This has come up a couple of times in conversations since the Almanac was published, speci… --- ## [Lazy-loaded LCP images by WordPress Version](https://discuss.httparchive.org/t/lazy-loaded-lcp-images-by-wordpress-version/2455)
**Author:** [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Replies:** 2\ **Last updated:** [October 17, 2022, 3:30pm UTC](https://discuss.httparchive.org/t/lazy-loaded-lcp-images-by-wordpress-version/2455 "2022-10-17T15:30:39Z")
Of the LCP elements in the September 2022 crawl that were IMG elements with native lazy loading (loading=lazy), How many were from WordPress and what was the WordPress version breakdown? WP Version Count “6.0.2” … --- ## [How does Drupal do so well on Core Web Vitals?](https://discuss.httparchive.org/t/how-does-drupal-do-so-well-on-core-web-vitals/2307)
**Author:** [@adamsilverstein](https://discuss.httparchive.org/u/adamsilverstein)\ **Replies:** 2\ **Last updated:** [February 5, 2022, 12:48am UTC](https://discuss.httparchive.org/t/how-does-drupal-do-so-well-on-core-web-vitals/2307 "2022-02-05T00:48:47Z")
I’ve been looking at the core web vitals report to see how Content Management Systems are performing as part of the WordPress Performance team. WordPress has a relatively poor pass rate, with only ~22% of sites “Having G… --- ## [Correlating Lighthouse scores to page-level CrUX data](https://discuss.httparchive.org/t/correlating-lighthouse-scores-to-page-level-crux-data/2164)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 6\ **Last updated:** [July 20, 2021, 2:33pm UTC](https://discuss.httparchive.org/t/correlating-lighthouse-scores-to-page-level-crux-data/2164 "2021-07-20T14:33:40Z")
The pages dataset contains a new field in the JSON-encoded HAR payloads: \_CrUX. As you can guess, this represents the real-user field data from the Chrome UX Report (CrUX) dataset. This presents a really interesting oppo… --- ## [Representative URLs for common unusual markup for investigation](https://discuss.httparchive.org/t/representative-urls-for-common-unusual-markup-for-investigation/1706)
**Author:** [@briankardell](https://discuss.httparchive.org/u/briankardell)\ **Replies:** 8\ **Last updated:** [August 28, 2019, 1:22am UTC](https://discuss.httparchive.org/t/representative-urls-for-common-unusual-markup-for-investigation/1706 "2019-08-28T01:22:59Z")
As part of the almanac efforts, we have redone the way we collect information about what elements are used - we’re now actively using the DOM tree and simply iterating and collecting the .tagNames of elements and increme… --- ## [What are the effects of the COVID-19 pandemic on web usage and experience?](https://discuss.httparchive.org/t/what-are-the-effects-of-the-covid-19-pandemic-on-web-usage-and-experience/1901)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 21\ **Last updated:** [July 2, 2020, 4:50pm UTC](https://discuss.httparchive.org/t/what-are-the-effects-of-the-covid-19-pandemic-on-web-usage-and-experience/1901 "2020-07-02T16:50:26Z")
The world has changed a lot in the past few months with the spread of the coronavirus. The web has been one area that has seen a dramatic shift, with so many people working and learning from home and using it to stay up … --- ## [Lighthouse Options](https://discuss.httparchive.org/t/lighthouse-options/2333)
**Author:** [@Arraa](https://discuss.httparchive.org/u/Arraa)\ **Replies:** 2\ **Last updated:** [September 24, 2022, 9:18am UTC](https://discuss.httparchive.org/t/lighthouse-options/2333 "2022-09-24T09:18:52Z")
Hello! Firstly, thanks for the awesome project. I’m interested about details how Lighthouse is used in Loading Speed-section, and I could not find the answer to the question I was thinking about. I am currently working … --- ## [Is the median page load time for the top 'x' pages tracked anywhere?](https://discuss.httparchive.org/t/is-the-median-page-load-time-for-the-top-x-pages-tracked-anywhere/2342)
**Author:** [@Nooshu](https://discuss.httparchive.org/u/Nooshu)\ **Replies:** 4\ **Last updated:** [April 8, 2022, 6:08pm UTC](https://discuss.httparchive.org/t/is-the-median-page-load-time-for-the-top-x-pages-tracked-anywhere/2342 "2022-04-08T18:08:21Z")
If I wanted to compare a websites own RUM data vs ‘global’ RUM data e.g. page load, is there anywhere that tracks this over time? e.g. median onload for March 2022. It’s a little like this report, but I’d like to know w… --- ## [Chrome Image Lazy Loading - Sites Already Using it on Week 1!](https://discuss.httparchive.org/t/chrome-image-lazy-loading-sites-already-using-it-on-week-1/1707)
**Author:** [@paulcalvano](https://discuss.httparchive.org/u/paulcalvano)\ **Replies:** 5\ **Last updated:** [March 9, 2020, 2:02pm UTC](https://discuss.httparchive.org/t/chrome-image-lazy-loading-sites-already-using-it-on-week-1/1707 "2020-03-09T14:02:30Z")
Earlier this year the Chrome team announced plans to support lazy loading natively in the browser. The plan was to add a loading attribute in both \ and \ elements. Chrome 75 included it behind a feature flag … --- ## [JavaScript Library Detection](https://discuss.httparchive.org/t/javascript-library-detection/955)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 19\ **Last updated:** [October 26, 2018, 6:49pm UTC](https://discuss.httparchive.org/t/javascript-library-detection/955 "2018-10-26T18:49:22Z")
Starting in the April 15, 2017 crawl, har tables will include a \_third-parties custom metric. This includes name and version information for ~100 of the most popular JavaScript libraries used on the web. This enables an … --- ## [Where to find http headers in the http archive datasets](https://discuss.httparchive.org/t/where-to-find-http-headers-in-the-http-archive-datasets/2345)
**Author:** [@bizzbyster](https://discuss.httparchive.org/u/bizzbyster)\ **Replies:** 2\ **Last updated:** [April 7, 2022, 7:32pm UTC](https://discuss.httparchive.org/t/where-to-find-http-headers-in-the-http-archive-datasets/2345 "2022-04-07T19:32:28Z")
Can someone point me to where in the data set I can find the response headers that would include Link and “nopush” attributes mentioned here? Thanks! --- ## [Strange values for total number of React origins](https://discuss.httparchive.org/t/strange-values-for-total-number-of-react-origins/2338)
**Author:** [@DanShappir](https://discuss.httparchive.org/u/DanShappir)\ **Replies:** 5\ **Last updated:** [March 31, 2022, 7:56am UTC](https://discuss.httparchive.org/t/strange-values-for-total-number-of-react-origins/2338 "2022-03-31T07:56:00Z")
I’m using the excellent Core Web Vitals Technology Report that @rviscomi created. When looking at the total number of origins for React I see a significant drop in July 2020 from 385K to 200K. Likewise in July 2021 I see… --- ## [How are news sites published?](https://discuss.httparchive.org/t/how-are-news-sites-published/2331)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 3\ **Last updated:** [March 16, 2022, 10:21am UTC](https://discuss.httparchive.org/t/how-are-news-sites-published/2331 "2022-03-16T10:21:17Z")
Two questions: how can we identify which websites serve news content, and what CMSs do each of these sites use? To identify a news site, we could use heuristics like the “news” keyword somewhere in the URL, title, or me… --- ## [Annual / seasonal trends in HTTP Archive data](https://discuss.httparchive.org/t/annual-seasonal-trends-in-http-archive-data/2320)
**Author:** [@pragmaticpat](https://discuss.httparchive.org/u/pragmaticpat)\ **Replies:** 4\ **Last updated:** [March 9, 2022, 12:57pm UTC](https://discuss.httparchive.org/t/annual-seasonal-trends-in-http-archive-data/2320 "2022-03-09T12:57:56Z")
I saw a reduction in the number of sites between January and February in 2022 for the technologies I’m watching, and so I took a look at the overall number of sites sampled in those months to determine whether the reduce… --- ## [Is the web changing as a result of IE's EOL?](https://discuss.httparchive.org/t/is-the-web-changing-as-a-result-of-ies-eol/2301)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 0\ **Last updated:** [January 27, 2022, 4:30am UTC](https://discuss.httparchive.org/t/is-the-web-changing-as-a-result-of-ies-eol/2301 "2022-01-27T04:30:33Z")
Inspired by The baseline for web development in 2022 article, which leads with: The biggest change that happened in 2021 ~ The retirement of Internet Explorer That got me wondering if there are any observable effects… --- ## [Who are the top RUM analytics providers?](https://discuss.httparchive.org/t/who-are-the-top-rum-analytics-providers/15)
**Author:** [@igrigorik](https://discuss.httparchive.org/u/igrigorik)\ **Replies:** 26\ **Last updated:** [January 12, 2022, 4:59pm UTC](https://discuss.httparchive.org/t/who-are-the-top-rum-analytics-providers/15 "2022-01-12T16:59:40Z")
--- ## [What is the addressable market of active web development?](https://discuss.httparchive.org/t/what-is-the-addressable-market-of-active-web-development/2294)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 1\ **Last updated:** [January 7, 2022, 6:54pm UTC](https://discuss.httparchive.org/t/what-is-the-addressable-market-of-active-web-development/2294 "2022-01-07T18:54:10Z")
Inspired by Malte’s tweet: What percent of websites changed their tech stack in a given month/year? --- ## [December 2022-specific decrease in data?](https://discuss.httparchive.org/t/december-2022-specific-decrease-in-data/2290)
**Author:** [@pragmaticpat](https://discuss.httparchive.org/u/pragmaticpat)\ **Replies:** 2\ **Last updated:** [January 4, 2022, 3:04pm UTC](https://discuss.httparchive.org/t/december-2022-specific-decrease-in-data/2290 "2022-01-04T15:04:42Z")
I’m researching web / javascript frameworks month-to-month in the httparchive.technologies data set, and noticed a sizable decrease in data between November 2021 and December 2021 this past year for all “app” values. T… --- ## [How can I run a query against a data subset?](https://discuss.httparchive.org/t/how-can-i-run-a-query-against-a-data-subset/2274)
**Author:** [@imkevdev](https://discuss.httparchive.org/u/imkevdev)\ **Replies:** 0\ **Last updated:** [November 24, 2021, 10:58pm UTC](https://discuss.httparchive.org/t/how-can-i-run-a-query-against-a-data-subset/2274 "2021-11-24T22:58:34Z")
Is there an “easy” way to run a query on a subset of the data without re-writing the query. For example, we know that 94.4% of websites use at least one third-party resource (source: Third Parties | 2021 | The Web Alman… --- ## [Usage of Web Monetization 2019 vs 2020](https://discuss.httparchive.org/t/usage-of-web-monetization-2019-vs-2020/2061)
**Author:** [@zcorpan](https://discuss.httparchive.org/u/zcorpan)\ **Replies:** 7\ **Last updated:** [November 22, 2021, 4:37pm UTC](https://discuss.httparchive.org/t/usage-of-web-monetization-2019-vs-2020/2061 "2021-11-22T16:37:49Z")
I wanted to check the adoption rate of Web Monetization, so I ran two queries in httparchive. Results 2019-10-01: 134 pages out of 4,325,232 = 0.0031% 2020-10-01: 415 pages out of 5,506,819 = 0.0075% So, very few page… --- ## [Who is driving dns-prefetch adoption?](https://discuss.httparchive.org/t/who-is-driving-dns-prefetch-adoption/2263)
**Author:** [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Replies:** 3\ **Last updated:** [November 12, 2021, 8:21am UTC](https://discuss.httparchive.org/t/who-is-driving-dns-prefetch-adoption/2263 "2021-11-12T08:21:17Z")
Posing a challenge to all the data analysts in the room :cowboy\_hat\_face: In the 2021 Resource Hints chapter of the Web Almanac (coming soon!) author @imkevdev says: The most widely used resource hint is dns-prefetch … --- ## [Growth of the Web in 2020](https://discuss.httparchive.org/t/growth-of-the-web-in-2020/2029)
**Author:** [@paulcalvano](https://discuss.httparchive.org/u/paulcalvano)\ **Replies:** 0\ **Last updated:** [September 29, 2020, 6:38pm UTC](https://discuss.httparchive.org/t/growth-of-the-web-in-2020/2029 "2020-09-29T18:38:35Z")
In a blog post on paulcalvano.com, I shared some insights into the growth of the web, as measured by an increase in the number of origins in the Chrome User Experience Report. This post shares the SQL queries and data I … --- ## [Is there an explanation of the columns](https://discuss.httparchive.org/t/is-there-an-explanation-of-the-columns/2231)
**Author:** [@nstringham](https://discuss.httparchive.org/u/nstringham)\ **Replies:** 8\ **Last updated:** [October 19, 2021, 7:56pm UTC](https://discuss.httparchive.org/t/is-there-an-explanation-of-the-columns/2231 "2021-10-19T19:56:41Z")
I am going through httparchive.summary\_pages.2021\_09\_01\_mobile in BigQuery preview and I don’t know what some of the columns mean. Is there documentation somewhere that describes each column? --- ## [Lighthouse scores as predictors of page-level CrUX data](https://discuss.httparchive.org/t/lighthouse-scores-as-predictors-of-page-level-crux-data/2232)
**Author:** [@brendankenny](https://discuss.httparchive.org/u/brendankenny)\ **Replies:** 2\ **Last updated:** [October 15, 2021, 9:18pm UTC](https://discuss.httparchive.org/t/lighthouse-scores-as-predictors-of-page-level-crux-data/2232 "2021-10-15T21:18:40Z")
Inspired by Rick’s earlier thread “Correlating Lighthouse scores to page-level CrUX data”, I started looking at the question of how to evaluate the Lighthouse score as a predictor of field data. As discussed in that thre… [Previous page](https://discuss.httparchive.org/c/analysis/1.md) [Next page](https://discuss.httparchive.org/c/analysis/1.md?page=2)