Question from the Twitterverse:
1 Like
I decided to take a stab that this by aggregating the mimeTypes in the Desktop and Mobile requests tables where type="script". You can see the query for this below:
SELECT mimeType, count(*) freq
FROM (
SELECT mimeType
FROM `httparchive.runs.2017_10_15_requests`
WHERE type = "script"
UNION ALL
SELECT mimeType
FROM `httparchive.runs.2017_10_15_requests_mobile`
WHERE type = "script"
)
GROUP BY mimeType
ORDER BY freq DESC
Overall there are 353 mime types detected, with the first 6 accounting for 99.53%. You can see the most frequent mimeType’s along with the percent and cumulative percent below:
The requests table also includes the extension, and we can analyze this further by adding it into the query:
SELECT mimeType, ext, count(*) freq
FROM (
SELECT mimeType, ext
FROM `httparchive.runs.2017_10_15_requests`
WHERE type = "script"
UNION ALL
SELECT mimeType, ext
FROM `httparchive.runs.2017_10_15_requests_mobile`
WHERE type = "script"
)
GROUP BY mimeType, ext
ORDER BY freq DESC
The full list of all 353 mime types is below, but you’ll notice that the 5th most frequent one is null. It seems there are quite a few sites loading JavaScript without a mimetype - which could be problematic if you plan to restrict based on mime type…
application/javascript
text/javascript
application/x-javascript
application/json
text/html
text/plain
application/octet-stream
text/json
text/js
text/x-js
text/x-json
application/octet-stream, application/javascript
application/opensearchdescription+xml
binary/octet-stream
application/zip
application/octet-stream, text/javascript
application/ecmascript
text/javascript, charset=utf-8
application/vnd.api+json
application/javascript, application/javascript
unknown/unknown
application/jsonp
application/vnd.softswiss.v1+json
text/ecmascript
application/vnd.maxmind.com-country+json
application/hal+json
application/x-amz-json-1.1
application/javascript, text/javascript
application / json
application/vnd.maxmind.com-city+json
text/JavaScript
application/json, application/json
Application/js
application/vnd.contentful.delivery.v1+json
application/x-javascript, application/x-javascript
application/js
application/octetstream
application/javascript; charset=utf-8
application/x-javascript, text/javascript
application/x-httpd-php
application/x-javascript; charset=utf-8
application/x-gzip
text/javascript; charset=UTF-8
json
application/x-javascript, application/javascript
Content-type: text/javascript
application/octet-stream, application/x-javascript
Content-Type: application/json
application/x-js
text/plain charset=UTF-8
application/binary
application/content+json
text/javascript charset=utf-8
text/x-javascript
plain/text
application/x-json
text/x-c
text/HTML
text/javascript
js
text/plain, text/javascript
application/json, charset=utf-8
application/json charset=UTF-8
application/unknown
application/vnd.maxmind.com-error+json
application/json,charset=utf-8
application/vnd.rbgemc+json
application/octet-stream, application/json
application/vnd.cartera.ocapi+json
application/x-www-form-urlencoded
application/json; charset=utf-8
application/json, text/javascript
text/Javascript
application/javascript; charset=UTF-8
none
text/html, application/json
application/x_javascript
text/html,application/json
aplication/json
application/vnd.geo+json
application/javascriptapplication/x-javascript
application/x-amz-json-1.0
baiduApp/json
script/javascript
text/javascript; charset=utf-8
application/x-octet-stream
inconnu/inconnu
application/x-elc
application/json
application/vnd.kafka.v1+json
application/json charset=UTF-8, text/plain
text/javscript
application/json;charset=UTF-8
text/javascript;charset=UTF-8
text/html, application/javascript
text/html; charset=UTF-8
type/javascript
text/x-c++
application/json; charset=UTF-8
text/html
text/scriptlet
application/JSON
text/jscript
text/javascript+json
text/html; charset=utf-8
application/javascriptapplication/x-javascripttext/javascript
appsication/x-javascript
dynamo-internal/forbidden
application/vnd.swm.v3+json
application/x-jaascript
json/application
application
application/javascript, text/html
Application/json
application/ld+json
text/javascript charset=UTF-8
application/javascript, application/x-javascript
text/txt
application/vnd.playlist.vrt.be.noa_1.0+json
application/x-j
jsonp
application/json charset=utf-8
application:json
application/json-rpc
Application/Json
application/x-javascript,application/x-javascript
application/x-directory
application/json,content-encoding:gzip,vary:accept-encoding
text/javascript;charset=utf-8
application/x-JavaScript
script
application/x-javascripts
application/vnd.gramedia.v3+json
application/x-download
application/vnd.mangahigh.api-v1+json
text/javascript,application/json
text/javascript charset=utf8
application/vnd.epg.vrt.be.onairs_1.2+json
application/api-v3+json
text/0.4/hammer.min.js
unknown
application/problem+json
application/download
Application/x-javascript
inode/x-empty
js/x-javascript
application/javascript;charset=utf-8
application/javascript;charset=UTF-8
application/vnd.novomind.iq-v3.0+json
text/JSON
application/force-download
application/x-javacript
application/jsonrequest
JSON
application/vnd.rba.auctions.v1+json
text/x-pascal
application/json;charset=ISO-8859-1
application/vnd.active-profile.v1+json
application/x-javascript,application/json
application/vnd.data-layer.v1+json
application/vnd.tiaa.personalization-data-store-rs-v1.0+json
text
application/x/javascript
application/transit+json
application/vnd.stream-cz.api+hal+json
application/x-
text/script
application/x-unknown-content-type
x-application/javascript
text/javascript, text/javascript
application/vnd.maxmind.com-insights+json
application/vnd.rb.uim-v8+json
text/javasript
text/x-handlebars-template
text/javascript, application/xml
application/x-web-app-manifest+json
application/postscript
TEXT/JAVASCRIPT
"text/javascript
application/truelocal-1.0+json
application/x-javascript, text/html
text/json-comment-filtered
application/javascript,application/x-javascript,text/javascript
application/Javascript
application/manifest+json
application/JavaScript
application/vnd.epg.vrt.be.channel_2.0+json
javascript/x-javascript
application-json
application/vnd.privategriffe+json
application/vnd.mason+json
applicaton/json
text/x-asm
application/x-httpd-php5
application/vnd.nrk.psapi+json
accplication/x-javascript
datalication/x-javascript
application/vnd.pxs.index.v1+json
application/vnd.epg.vrt.be.channels_1.1+json
application/vnd.radio-canada.neuro+json
application/vnd.epg.vrt.be.onairs_1.0+json
application/Json
text/Actionscript
application/opensearchdescription+xml; charset=utf-8
fcgid-script
application/javascript+jsonp
charset=UTF-8
Application/javascript
application\\json
text\\javascript
application/x-javascript,text/html
application/x-view-source
text/javacript
application/javascript.
application/vnd.shureeu.v1+json
application/vnd.buckets+json
text/x-script.perl
application/gzip
x-mapp-php5
application/vnd.rba.search.v1+json
application/vnd.qiwi.sso-v1+json
application/x-javascript;charset=UTF-8
application/vnd.wg.cds_api+json
application/javascript,Charset=UTF-8
text-javascript
text/javascript;charset=ISO-8859-1
application/x-script
applecation/json
application/vnd.de.zdf.v1.0+json
application/x-javascript charset=utf-8
x-javascript
text/javascipt
ext/javascript
application/vnd.sequoia+json
"text/javascript "
application/empty
.js
text/csv
text/plain, application/javascript
application/rsvpjson-v1+json
text/html; charset=iso-8859-1
application/vnd.kafka.v2+json
application/javascript, */*
text/javascript
application/AngularJS
javascript
x-json
application/vnd.session-service+json
application/vnd.com.shoebuy.v1+json
application/vnd.weather.vrt.be.observations_1.0+json
text/javascript charset="utf-8"
application/vnd.rb.uim-v6+json
application/vnd.ez.api.Root+json
'text/javascript'
application/vnd.maxmind.com-city-isp-org+json
application/vnd.digipost-v2+json
text/x-netuix-json-comment-filtered
application/json, text/plain, */*
application/x-perl
application/vnd.traffic.vrt.be.traffic_jam_length_1.0+json
​application/javascript
"application/x-javascript"
text/json charset=utf-8
application/x-javascript charset="utf-8"
application/json,text/x-json,application/jsonrequest,text/json
application/x-javasc
text/x-matlab
application/javascrit
application/x-msdownload
text/x-script.ruby
txt/javascript
application/javascript,text/html
application/vnd.allegro.public.v1+json
application/javascript, charset=utf-8
application/JSONP
tk/relay
text/json, application/json
application/x-java-pack200
UTF-8
application/vnd.collection+json
application/vnd.music.vrt.be.songs_2.0+json
application/vnd.countries+json
application/vnd.preferences+json
<module 'json' from '/usr/local/lib/python3.4/json/__init__.py'>
application/x-javascript exts=js
json/json
application/vnd.graphite.v1+json
application/vnd.mywebgrocer.account-entry+json
application/vnd.mywebgrocer.stores+json
jsonp/plain
application/com.bclc.csg.lotto.results+json
appplication/json
application/vnd.cft-data.v1.8+json
application/vnd.sumall.platforms+json
multipart/byteranges
application/json, text/plain
Application/JSON
application/vnd.art19.v0+json
text/min.js
text/json, charset=utf-8
application/lance+json
type/JSON
application/vnd.blueapron.com.v20150501+json
{'Content-Type': 'application/json'}
application/vnd.lotsys.itf.jackpot-1+json
appliction/json
application/vnd.com.dominos.ecommerce.store-locator.response+json
Content-Type:application/javascript
text/json;charset=UTF-8
application/vnd.vmware.horizon.manager.error+json
86400
application/vnd.live-list+json
application/rsvpjson-v3+json
php5-script
application/javascript, application/x-javascript, text/javascript
application/rsvpjson-v2+json
text/html, text/plain, text/plain, text/plain, text/plain, text/plain
application: json
text/json
application/vnd.villaday.v2+json
application/json, application/javascript
application/x-javascript;charset=utf-8
text/javascript, charset=UTF-8
text/html, text/plain, text/plain
application/json,Charset=UTF-8
text/x-csrc
application/geojson
application/json, text/plain, text/plain
text/html, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain
"text/javascript"
text/html, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain
application/x-javascript; charset=UTF-8
application/vnd.otto.flyout+json
application/x-javascript; charset=gb2312
text/html, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain
application/octet-stream, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain
php-script
application/vnd.swacorp.com.air-reservations.last-bookable-date-v1.0+json
text/x-json; charset=UTF-8
.json
application/vnd.overstock-v1+json
application/javascript, text/javascript, text/javascript, text/javascript, text/javascript
appliaction/json
text/html, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain, text/plain
application/vnd.autorei.sponsoredProductList+json
application/javascript, text/javascript, text/javascript
application/vnd.cm.messages_version_1.0+json
content-type: application/json
application/json, text/javascript, */*
application/octet-stream, text/plain, text/plain, text/plain, text/plain, text/plain
text/html, text/plain, text/plain, text/plain, text/plain
text/json,charset=utf8
Exploring the ~ 2.25% of all scripts in the HTTP Archive that had no mimeType associated with them, I found they mainly fell into 3 categories:
- JavaScript from Third Party Ads
- JS or JSON responses from social sharing functionality such as twitter, pinterest, etc.
- 3rd party beacons that returned 0 bytes
2 Likes
Thank you both! Just as a bit of context: We would like to restrict script loads to just JavaScript mime types for security reasons. Sadly this is not really possible, as you see … We have already explicitly blocked some mime types before: https://github.com/whatwg/fetch/pull/379 and would like to do that again for more uncommon mime types.
The Firefox telemetry looks like this: https://mzl.la/2A03rkD. We currently don’t have the most recent data, because the counter was automatically disabled.
2 Likes