# Websites with cache-control: no-store, by CDN

**URL:** <https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619>\
**Category:** Analysis\
**Created:** [August 30, 2023, 3:24pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619 "2023-08-30T15:24:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![aaronpeters](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/aaronpeters/32/1723_2.png) [@aaronpeters](https://discuss.httparchive.org/u/aaronpeters)\
**Post date:** [August 30, 2023, 3:24pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/1 "2023-08-30T15:24:01Z")

</div>

I recently queried the HTTP Archive to get the number of websites (pages) that serve the HTML with response header `cache-control` contains `no-store` .

My query:

```sql
SELECT
  _TABLE_SUFFIX AS client,
  count(DISTINCT pageid) AS pages,
  pages_total,
  count(DISTINCT pageid) / pages_total AS pages_pct,
FROM
  `httparchive.summary_requests.2021_08_01_*`
JOIN
  (SELECT _TABLE_SUFFIX, COUNT(DISTINCT pageid) AS pages_total FROM `httparchive.summary_requests.2021_08_01_*` GROUP BY _TABLE_SUFFIX)
USING (_TABLE_SUFFIX)
WHERE
  LOWER(resp_cache_control) LIKE '%no-store%' AND
  firstHTML
GROUP BY
  _TABLE_SUFFIX,
  pages_total

```

Now, I’d like take a next step and get the number of pages with that header, by CDN.  
Say the total is 2,000,000 pages with that header’s directive on mobile, how many of those are served by Akamai, Cloudflare, Fastly, etc?

But ideally I can get the % of pages that each CDN serves with that header’s directive (a bfcache killer ☹ ), for example:

Akamai:

- Total pages: 200,000
- Pages with `no-store`: 100,000
- % pages with `no-store`: 50%

My SQL is rusty at best and I can’t get it to work in BigQuery.  
Anybody have some pointers for me? Txs !

---

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [September 7, 2023, 3:06pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/3 "2023-09-07T15:06:34Z")

</div>

Hi @aaronpeters 👋

I took a stab at this using the new `all.pages` table for CDN detection and `all.requests` table for `no-store` detection. My general approach is to use `WITH` subqueries for each of these pieces, then bring them together with a `JOIN` in the final `SELECT` statement.

```sql
WITH no_store AS (
  SELECT
    page
  FROM
    `httparchive.all.requests`,
    UNNEST(response_headers) AS header
  WHERE
    date = '2023-08-01' AND
    client = 'mobile' AND
    is_root_page AND
    is_main_document AND
    LOWER(header.name) = 'cache-control' AND
    REGEXP_CONTAINS(header.value, r'no-store')
),

cdns AS (
  SELECT
    rank,
    page,
    JSON_VALUE(summary, '$.cdn') AS cdn
  FROM
    `httparchive.all.pages`
  WHERE
    date = '2023-08-01' AND
    client = 'mobile' AND
    is_root_page AND
    JSON_VALUE(summary, '$.cdn') != ''
)

SELECT
  cdn,
  COUNTIF(no_store.page IS NOT NULL) AS pages_with_no_store,
  COUNT(0) AS total_pages,
  COUNTIF(no_store.page IS NOT NULL) / COUNT(0) AS pct_no_store
FROM
  cdns
LEFT JOIN
  no_store
USING
  (page)
GROUP BY
  cdn
HAVING
  total_pages >= 1000
ORDER BY
  total_pages DESC

```

Note that the we’re detecting CDNs based on WebPageTest.

Here are the results for CDNs with 1k+ sites, ordered by popularity:

| cdn | pages\_with\_no\_store | total\_pages | pct\_no\_store |
| --- | --- | --- | --- |
| Cloudflare | 391,675 | 2,548,966 | 15.4% |
| Google | 139,629 | 1,099,485 | 12.7% |
| Fastly | 12,681 | 363,678 | 3.5% |
| Amazon CloudFront | 66,242 | 297,571 | 22.3% |
| Akamai | 43,628 | 115,532 | 37.8% |
| Automattic | 5 | 105,874 | 0.0% |
| Sucuri Firewall | 8,157 | 53,357 | 15.3% |
| Vercel | 6,326 | 47,304 | 13.4% |
| Incapsula | 11,806 | 38,333 | 30.8% |
| Microsoft Azure | 4,926 | 20,522 | 24.0% |
| CDN | 1,173 | 19,184 | 6.1% |
| Cloudflare, Google | 4,858 | 16,901 | 28.7% |
| OVH CDN | 3,669 | 14,450 | 25.4% |
| Netlify | 282 | 13,806 | 2.0% |
| Cloudflare, Fastly | 665 | 13,434 | 5.0% |
| Amazon CloudFront, Cloudflare | 1,123 | 12,282 | 9.1% |
| Highwinds | 5,875 | 11,414 | 51.5% |
| Zenedge | 10,353 | 10,700 | 96.8% |
| GoCache | 8 | 8,208 | 0.1% |
| section.io | 4,041 | 7,820 | 51.7% |
| Azion | 118 | 6,581 | 1.8% |
| Alibaba | 595 | 5,531 | 10.8% |
| QUIC.cloud | 405 | 4,214 | 9.6% |
| Fastly, Google | 141 | 3,551 | 4.0% |
| Tencent | 170 | 2,698 | 6.3% |
| BunnyCDN | 27 | 2,528 | 1.1% |
| CDNetworks | 314 | 1,338 | 23.5% |

Here’s another view of the top 10 CDNs by % `no-store`:

| cdn | pages\_with\_no\_store | total\_pages | pct\_no\_store |
| --- | --- | --- | --- |
| Zenedge | 10,353 | 10,700 | 96.8% |
| section.io | 4,041 | 7,820 | 51.7% |
| Highwinds | 5,875 | 11,414 | 51.5% |
| Akamai | 43,628 | 115,532 | 37.8% |
| Incapsula | 11,806 | 38,333 | 30.8% |
| Cloudflare, Google | 4,858 | 16,901 | 28.7% |
| OVH CDN | 3,669 | 14,450 | 25.4% |
| Microsoft Azure | 4,926 | 20,522 | 24.0% |
| CDNetworks | 314 | 1,338 | 23.5% |
| Amazon CloudFront | 66,242 | 297,571 | 22.3% |
| Cloudflare | 391,675 | 2,548,966 | 15.4% |
| Sucuri Firewall | 8,157 | 53,357 | 15.3% |

Does this answer your question?

---

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [September 7, 2023, 3:27pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/4 "2023-09-07T15:27:40Z")

</div>

Also, here’s a sample of 100 of the most popular pages to use `no-store`:

```sql
SELECT
  rank,
  cdn,
  page
FROM
  cdns
JOIN
  no_store
USING
  (page)
ORDER BY
  rank,
  NET.REG_DOMAIN(page)
LIMIT
  100

```

| rank | cdn | page |
| --- | --- | --- |
| 1,000 | Akamai | [https://www.accuweather.com/](https://www.accuweather.com/) |
| 1,000 | Akamai | [https://www.ad.nl/](https://www.ad.nl/) |
| 1,000 | Akamai | [https://m.alibaba.com/](https://m.alibaba.com/) |
| 1,000 | Cloudflare | [https://altibbi.com/](https://altibbi.com/) |
| 1,000 | Akamai | [https://sellercentral.amazon.com/](https://sellercentral.amazon.com/) |
| 1,000 | Fastly | [https://ameblo.jp/](https://ameblo.jp/) |
| 1,000 | Cloudflare | [https://www.ancestry.com/](https://www.ancestry.com/) |
| 1,000 | Cloudflare | [https://bbs.animanch.com/](https://bbs.animanch.com/) |
| 1,000 | Cloudflare | [https://www.arbada.com/](https://www.arbada.com/) |
| 1,000 | Cloudflare | [https://m.beboo.ru/](https://m.beboo.ru/) |
| 1,000 | Cloudflare | [https://www.bet365.com/](https://www.bet365.com/) |
| 1,000 | Cloudflare | [https://br.betano.com/](https://br.betano.com/) |
| 1,000 | Amazon CloudFront | [https://www.bikewale.com/](https://www.bikewale.com/) |
| 1,000 | Cloudflare | [https://in.bookmyshow.com/](https://in.bookmyshow.com/) |
| 1,000 | Fastly | [https://www.buzzfeed.com/](https://www.buzzfeed.com/) |
| 1,000 | Cloudflare | [https://www.canva.com/](https://www.canva.com/) |
| 1,000 | Amazon CloudFront | [https://coinmarketcap.com/](https://coinmarketcap.com/) |
| 1,000 | Akamai | [https://www.coppel.com/](https://www.coppel.com/) |
| 1,000 | Akamai | [https://link.coupang.com/](https://link.coupang.com/) |
| 1,000 | Akamai | [https://www.creditkarma.com/](https://www.creditkarma.com/) |
| 1,000 | Amazon CloudFront, Cloudflare | [https://www.curseforge.com/](https://www.curseforge.com/) |
| 1,000 | Cloudflare | [https://www.dagbladet.no/](https://www.dagbladet.no/) |
| 1,000 | Incapsula | [https://hiroba.dpoint.docomo.ne.jp/](https://hiroba.dpoint.docomo.ne.jp/) |
| 1,000 | Cloudflare | [https://www.dogdrip.net/](https://www.dogdrip.net/) |
| 1,000 | Cloudflare | [https://doublelist.com/](https://doublelist.com/) |
| 1,000 | Cloudflare | [https://dualeotruyenz.com/](https://dualeotruyenz.com/) |
| 1,000 | Akamai | [https://www.elcorteingles.es/](https://www.elcorteingles.es/) |
| 1,000 | Cloudflare | [https://pl.escort.club/](https://pl.escort.club/) |
| 1,000 | Akamai | [https://www.espncricinfo.com/](https://www.espncricinfo.com/) |
| 1,000 | Akamai | [https://www.etsy.com/](https://www.etsy.com/) |
| 1,000 | Akamai | [https://www.expressen.se/](https://www.expressen.se/) |
| 1,000 | Facebook | [https://web.facebook.com/](https://web.facebook.com/) |
| 1,000 | Facebook | [https://free.facebook.com/](https://free.facebook.com/) |
| 1,000 | Facebook | [https://www.facebook.com/](https://www.facebook.com/) |
| 1,000 | Facebook | [https://apps.facebook.com/](https://apps.facebook.com/) |
| 1,000 | Facebook | [https://mbasic.facebook.com/](https://mbasic.facebook.com/) |
| 1,000 | Facebook | [https://m.facebook.com/](https://m.facebook.com/) |
| 1,000 | Facebook | [https://business.facebook.com/](https://business.facebook.com/) |
| 1,000 | Cloudflare | [https://faphouse.com/](https://faphouse.com/) |
| 1,000 | Google | [https://www.finn.no/](https://www.finn.no/) |
| 1,000 | Cloudflare | [https://funpay.com/](https://funpay.com/) |
| 1,000 | Google | [https://play.google.com/](https://play.google.com/) |
| 1,000 | Google | [https://sites.google.com/](https://sites.google.com/) |
| 1,000 | Google | [https://docs.google.com/](https://docs.google.com/) |
| 1,000 | Google | [https://one.google.com/](https://one.google.com/) |
| 1,000 | Google | [https://mail.google.com/](https://mail.google.com/) |
| 1,000 | Google | [https://accounts.google.com/](https://accounts.google.com/) |
| 1,000 | Google | [https://news.google.com/](https://news.google.com/) |
| 1,000 | Google | [https://myactivity.google.com/](https://myactivity.google.com/) |
| 1,000 | Google | [https://translate.google.com/](https://translate.google.com/) |
| 1,000 | Cloudflare | [https://ww1.goojara.to/](https://ww1.goojara.to/) |
| 1,000 | Akamai | [https://www.hln.be/](https://www.hln.be/) |
| 1,000 | Amazon CloudFront | [https://www.imdb.com/](https://www.imdb.com/) |
| 1,000 | Amazon CloudFront | [https://m.imdb.com/](https://m.imdb.com/) |
| 1,000 | Cloudflare | [https://imhentai.xxx/](https://imhentai.xxx/) |
| 1,000 | Akamai | [https://www.incometax.gov.in/](https://www.incometax.gov.in/) |
| 1,000 | Fastly | [https://www.independent.co.uk/](https://www.independent.co.uk/) |
| 1,000 | Google | [https://m.indiamart.com/](https://m.indiamart.com/) |
| 1,000 | Akamai | [https://navbharattimes.indiatimes.com/](https://navbharattimes.indiatimes.com/) |
| 1,000 | Facebook | [https://www.instagram.com/](https://www.instagram.com/) |
| 1,000 | Cloudflare | [https://isaidub3.co/](https://isaidub3.co/) |
| 1,000 | Cloudflare | [https://jable.tv/](https://jable.tv/) |
| 1,000 | Akamai | [https://www.jiocinema.com/](https://www.jiocinema.com/) |
| 1,000 | Cloudflare | [https://www.jusbrasil.com.br/](https://www.jusbrasil.com.br/) |
| 1,000 | Cloudflare | [https://jut.su/](https://jut.su/) |
| 1,000 | Akamai | [https://www.kleinanzeigen.de/](https://www.kleinanzeigen.de/) |
| 1,000 | Cloudflare | [https://m.ladies.de/](https://m.ladies.de/) |
| 1,000 | Akamai | [https://www.latamairlines.com/](https://www.latamairlines.com/) |
| 1,000 | Akamai | [https://www.lavanguardia.com/](https://www.lavanguardia.com/) |
| 1,000 | Cloudflare | [https://www.learncbse.in/](https://www.learncbse.in/) |
| 1,000 | Akamai | [https://m.lightinthebox.com/](https://m.lightinthebox.com/) |
| 1,000 | Akamai | [https://today.line.me/](https://today.line.me/) |
| 1,000 | Microsoft Azure | [https://www.linkedin.com/](https://www.linkedin.com/) |
| 1,000 | Cloudflare | [https://litnet.com/](https://litnet.com/) |
| 1,000 | Akamai | [https://www.lowes.com/](https://www.lowes.com/) |
| 1,000 | Cloudflare | [https://en.luxuretv.com/](https://en.luxuretv.com/) |
| 1,000 | Amazon CloudFront | [https://select.mamastar.jp/](https://select.mamastar.jp/) |
| 1,000 | Cloudflare | [https://www.mangago.me/](https://www.mangago.me/) |
| 1,000 | Google | [https://www.marktplaats.nl/](https://www.marktplaats.nl/) |
| 1,000 | Cloudflare | [https://www.mediafire.com/](https://www.mediafire.com/) |
| 1,000 | Amazon CloudFront | [https://articulo.mercadolibre.com.mx/](https://articulo.mercadolibre.com.mx/) |
| 1,000 | Amazon CloudFront | [https://www.mercadolibre.com.mx/](https://www.mercadolibre.com.mx/) |
| 1,000 | Google | [https://homepage.mintnav.com/](https://homepage.mintnav.com/) |
| 1,000 | Cloudflare | [https://mobile-tracker-free.com/](https://mobile-tracker-free.com/) |
| 1,000 | Akamai | [https://www.moneycontrol.com/](https://www.moneycontrol.com/) |
| 1,000 | Cloudflare | [https://moviesda4.net/](https://moviesda4.net/) |
| 1,000 | Microsoft Azure | [https://www.msn.com/](https://www.msn.com/) |
| 1,000 | Akamai | [https://www.mundodeportivo.com/](https://www.mundodeportivo.com/) |
| 1,000 | Akamai | [https://www.myntra.com/](https://www.myntra.com/) |
| 1,000 | Cloudflare | [https://www.nairaland.com/](https://www.nairaland.com/) |
| 1,000 | Akamai | [https://n.news.naver.com/](https://n.news.naver.com/) |
| 1,000 | Akamai | [https://m.naver.com/](https://m.naver.com/) |
| 1,000 | Azion | [https://www.netshoes.com.br/](https://www.netshoes.com.br/) |
| 1,000 | Cloudflare | [https://olimpbet.kz/](https://olimpbet.kz/) |
| 1,000 | Cloudflare | [https://chat.openai.com/](https://chat.openai.com/) |
| 1,000 | Fastly | [https://www.paypal.com/](https://www.paypal.com/) |
| 1,000 | Amazon CloudFront | [https://picrew.me/](https://picrew.me/) |
| 1,000 | Cloudflare | [https://www.pixiv.net/](https://www.pixiv.net/) |
| 1,000 | Cloudflare | [https://www.rbc.ua/](https://www.rbc.ua/) |
| 1,000 | Amazon CloudFront | [https://www.redfin.com/](https://www.redfin.com/) |

---

<div class="post-metadata">

**Author:** ![aaronpeters](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/aaronpeters/32/1723_2.png) [@aaronpeters](https://discuss.httparchive.org/u/aaronpeters)\
**Post date:** [September 8, 2023, 6:20am UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/5 "2023-09-08T06:20:55Z")

</div>

Thanks @rviscomi you are the man! 🙏  
The query looks great and … so simple 😃

Thanks for including some data in your response, that is very nice.

First comments on the data:

1,099,485 pages served by Google :  
That sure is not companies who are using Google Cloud CDN for accelerating their website. I guess there are many, many Google web properties in HTTP Archive (.blogspot.com?)

“Cloudflare, Google”: does that signify multi-cdn?

What is “CDN”?

👏 Automattic looks fine in this table

---

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [September 8, 2023, 4:50pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/6 "2023-09-08T16:50:07Z")

</div>

Thanks @aaronpeters glad I can help.

> [@aaronpeters](#):
>
> What is “CDN”?

Here’s the WPT source where that detection is done, if interested:

> <https://github.com/catchpoint/WebPageTest.agent/blob/master/internal/optimization_checks.py#L218>

> [@aaronpeters](#):
>
> 1,099,485 pages served by Google :  
> That sure is not companies who are using Google Cloud CDN for accelerating their website. I guess there are many, many Google web properties in HTTP Archive (.blogspot.com?)

You’re right, when I look at the most common domain names behind the Google CDN, most of them are [blogspot.com](http://blogspot.com):

```sql
SELECT
  APPROX_TOP_COUNT(NET.REG_DOMAIN(page), 10)
FROM
  `httparchive.all.pages`
WHERE
  date = '2023-08-01' AND
  client = 'mobile' AND
  is_root_page AND
  JSON_VALUE(summary, '$.cdn') = 'Google'

```

| domain | pages |
| --- | --- |
| blogspot.com | 249,999 |
| business.site | 28,680 |
| softonic.com | 19,368 |
| negocio.site | 6,744 |
| eatbu.com | 5,908 |
| softonic.com.br | 3,253 |
| repl.co | 1,784 |
| netlify.app | 1,721 |
| choiceqr.com | 1,487 |
| play-cricket.com | 1,467 |

> [@aaronpeters](#):
>
> “Cloudflare, Google”: does that signify multi-cdn?

Yes IIRC that means it’s multi-CDN, with Cloudflare in front. @patmeenan can correct me if I’m wrong.

---

<div class="post-metadata">

**Author:** ![aaronpeters](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/aaronpeters/32/1723_2.png) [@aaronpeters](https://discuss.httparchive.org/u/aaronpeters)\
**Post date:** [September 8, 2023, 5:47pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/7 "2023-09-08T17:47:50Z")

</div>

Thanks @rviscomi and yes, I know of the WPT logic for CDN detection.  
That needs a clean up … NetDNA, HiberniaCDN and others should no longer be in there, and I wonder how WPT handles the case where Akamai has a CloudFront origin … which response header wins?  
Anyway …

---

<div class="post-metadata">

**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\
**Post date:** [September 8, 2023, 6:00pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/8 "2023-09-08T18:00:29Z")

</div>

None “win”. As Rick mentioned, both will get listed. There’s no harm in the old, dead CDN’s being there but always happy to accept PR’s.

---

<div class="post-metadata">

**Author:** ![aaronpeters](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/aaronpeters/32/1723_2.png) [@aaronpeters](https://discuss.httparchive.org/u/aaronpeters)\
**Post date:** [September 8, 2023, 7:36pm UTC](https://discuss.httparchive.org/t/websites-with-cache-control-no-store-by-cdn/2619/9 "2023-09-08T19:36:00Z")

</div>

Got it Pat.  
Will find the time for a PR soon.