How many resources have X Frame Options, Strict Transport Security, or Content Security Policy headers for web app security?

Analysis
1

Borrowing heavily from @stevesoudersorg’ post…

I wanted to use the HTTP Archive data to see how many resources have certain response headers intended for web application security.

This query tells us how many total requests were made in the November crawl:

    SELECT count(*)
    FROM [httparchive:runs.2013_11_15_requests]

result: 27,889,759 requests

Strict Transport Security (HSTS)

Purpose: to instruct browsers that your site is to only be accessed over https

    SELECT domain(url) as domain, count(*) as num
    FROM [httparchive:runs.2013_11_15_requests]
    WHERE lower(respOtherHeaders) contains "strict-transport-security"
    GROUP BY domain ORDER BY num desc

result: 830 sites with HSTS

830 / 27,889,759 = far less than one percent

If you are one of the valiant minority that has already implemented HSTS you should strongly consider requesting to have your site added to the list of sites which Chrome will always first visit using https. Instructions here and turnaround is usually quick. (Firefox also works off the same list, so all the more reason!)

X-Frame-Options

Purpose: to significantly reduce the risk of successful clickjacking attacks against your site.

    SELECT domain(url) as domain, count(*) as num
    FROM [httparchive:runs.2013_11_15_requests]
    WHERE lower(respOtherHeaders) contains "x-frame-options"
    GROUP BY domain ORDER BY num desc

result: 13,453 sites with X-Frame-Options

13,453 / 27,889,759 = far less than one percent

Content Security Policy

Purpose: to significantly reduce the risk of successful Cross Site Scripting (XSS) attacks against your site.

    SELECT domain(url) as domain, count(*) as num
    FROM [httparchive:runs.2013_11_15_requests]
    WHERE lower(respOtherHeaders) contains "content-security-policy"
    GROUP BY domain ORDER BY num desc

result: 69

69 / 27,889,759 = you don’t want to know

Yes, I know I didn’t include all the variants of the CSP headers that have been used. I don’t think it would change the outcome very much.

##Conclusion##

These numbers were expected to be low. At least now we have a baseline to compare against in future crawls.

In any case we have a lot of work to do to promote the use of headers like these and create a safer browsing experience online!

1 Like
2

Here’s the list of early adopters where Content Security Policy was found using the query:

SELECT domain(url) as domain, count(*) as num
FROM [httparchive:runs.2013_11_15_requests]
WHERE lower(respOtherHeaders) contains "content-security"
GROUP BY domain ORDER BY num desc

Congratulations to them for being at the forefront!

a-ads.com


addvocate.com
admagnet.net
aleo.pl

annuaire-inverse-france.com

b144.co.il
blockchain.info
borderlinx.com
bryk.pl
bullcloud.com
calomel.org
centrum24.pl
circuitsonline.net

defcon.org
digipost.no

dnb.no

fleep.io
fontello.com
forban.su


hackerone.com




kontent.com

lastpass.com
libravatar.org
likealyzer.com

mega.co.nz


mozillalabs.com
mts.ru

playboard.me

rbnz.govt.nz
savefront.com
sbwire.com
scalemodels.ru
searchfunmoods.com

simplemachines.org

tinywp.in
twimg.com
twitter.com
unibw.de
unitono.es
upf.co.il


w3.org
webhosting.co.uk
webintents.org
webropolsurveys.com
webshare.co
yliopistonapteekki.fi
zady.com

3

Is the query to calculate the total number of sites correct? It seems to return the total number of requests instead. I tried:

SELECT domain(url) as domain
    FROM [httparchive:runs.2013_11_15_requests]
    group by domain

This returns 358,638 sites.

4

Very nice increase in the number of sites using Content Security Policy!

Try this query:

SELECT domain(url) as domain, count(*) as num
FROM [httparchive:runs.2014_10_15_requests]
WHERE lower(respOtherHeaders) contains "content-security"
GROUP BY domain ORDER BY num desc

…to see the 336 sites using CSP now. That’s a big increase over the 69 sites shown in this thread using the Nov 2013 data! Nice job everybody!

5 
SELECT domain(url) as domain, count(*) as num
FROM [httparchive:runs.2015_10_15_requests]
WHERE lower(respOtherHeaders) contains "content-security"
GROUP BY domain ORDER BY num desc

One year later we have 3304 sites with Content Security Policy set up! That’s ten times what we had one year ago! Great job everyone!

6 
SELECT domain(url) as domain, count(*) as num
FROM [httparchive:runs.2016_11_01_requests]
WHERE lower(respOtherHeaders) contains "content-security"
GROUP BY domain ORDER BY num desc

One year later we have 6,276 sites with Content Security Policy set up! That’s twice what we had one year ago! Great job everyone!

8 
SELECT domain(url) as domain, count(*) as num 
FROM [httparchive:runs.2017_11_01_requests] 
WHERE lower(respOtherHeaders) contains "content-security" 
GROUP BY domain ORDER BY num desc

Another year later we have 13,215 sites with Content Security Policy set up! That’s more than twice what we had one year ago! Great job everyone!

1 Like