As of March 1 2016, both the desktop and mobile HTTP Archive tests are performed via Google Chrome: both use the latest (“evergreen”) stable build of Chrome and the mobile tests are done via Chrome’s device emulation (using Nexus 5 profile). This has been long in the making, and after months of hard work, validation, and bug fixing—big thanks to everyone who reported issues and helped fix them—it’s now the default for all runs moving forward.
What motivated the switch? In no particular order…
-
Internet Explorer 9 is no longer supported by Microsoft and does not provide the necessary support for the features and capabilities that are used by modern web applications. In the meantime, Chrome has now become the leader by market share.
-
We wanted to move towards the “evergreen browser” model: each run will use the latest stable build of Chrome, instead of being frozen in time. This should better reflect and capture the everyday experience as seen by the user.
-
Old mobile testing was limited because the tests ran on physical iOS devices: hence why we were only able to track the top 5K sites. In the meantime, the importance of mobile has grown in leaps and bounds (and that’s an understatement), and we’ve long wanted to expand our mobile coverage to match its desktop counterpart (top 500K), which we cannot do with the old approach.
This switch took us a long time because we did a lot of testing and manual verification to ensure that the switch to an emulated environment captures the right metrics—once again, big thanks to everyone that helped along the way.
As a side note, while working on the above, we’ve actually found a few cases where the old WebPageTest iPhone emulation rendered the site incorrectly, but our new emulated mode gets it right. That said, if you do find any issues, please let us know.
More great performance data is coming
- With the new infrastructure in place we’re now able to provide full HAR captures on every run (see quickstart guide, instructions for how to download it for local analysis, and BigQuery HAR dataset). These include content bodies for all “text” resources - e.g. JavaScript, CSS, and HTML.
- We’re investigating capture of Chrome’s feature histograms (i.e. data in chrome://histograms), which allows us to analyze which features are used by each site.
- We’re also looking at capturing Chrome execution traces (chrome://tracing), and DevTools timeline traces, which would give us an even deeper perspective into where and how the time is spent in the browser - e.g. scripts vs layout vs other tasks.
The HTTP Archive data is a tremendous resource. We’re excited to expand the breadth and depth of this free, open data set. In the meantime, if you spot any issues, or have any questions, just comment below.
p.s. @stevesoudersorg has additional coverage on the switch on his blog: HTTP Archive switches to Chrome.