I recently was looking at fullyloaded over time and I noticed that both desktop and mobile spiked with the 12/1 run. I am using the following syntax in BQ:
Select
integer(Avg(fullyLoaded))
FROM httparchive:runs.2017_12_01_pages
Where visualComplete > 0;
The following is a screenshot showing the spike starting in December -
Please note that my SQL is just a snippet. I am using quantiles to breakout the data between fastest, slowest, and overall. 
Has anyone come across this or does anyone know of a system change that would have caused this behavior?
Hmm, interesting. @patmeenan is this related to these?
Yep, the Facebook CDN peering in Palo Alto was showing upwards of 90% packet loss for several weeks. Any pages that included fbcdn static resources (like buttons, retargeting beacons, etc) got extremely delayed load times.
I believe it extended into the Dec 15th crawl as well and that one also got hit by a Level 3 backbone issue that impacted the west coast for a couple of days.
As far as I know, all of the issues were real production issues with Facebook and Level 3, isolated to the US west coast.
Patrick - Thank you for the clarification!!!
Hi Patrick - You mentioned that the issue crept into the 12/15 run as well. I just did a quick look at the latest page run and the responses have jumped even more dramatically:
Nov 17 --> 12312
Dec 12 --> 29711
Jan 18 --> 32629
Are you aware of any ongoing issues that would lead to this increased degradation?
What is “Dec 12”? Is that 17 (as in 2017) or the Dec 15 crawl? Are those the fully loaded times? How do the other metrics look?
There aren’t any ongoing issues that I’m aware of but I largely base my monitoring of the performance on how quickly it is going through the crawl and it’s running through at roughly normal speeds.
There’s one hidden graph in the main HA UI that shows times and it spiked for the Dec 1 crawl (load time only, not start render) but dropped back down for the Dec 15 and Jan 1 crawls.
Patrick - Sorry for the typo. I also should have been more descriptive. The periods above are from runs 11/1/17, 12/1/17, and 1/1/18 and represent fullyLoaded.
Let me rerun my extraction using the 1/1/18 for all key metrics I track and make sure I have not fat fingered something. I will then compare what what appears on the site and then circle back with my findings. Thanks.
Patrick - I reran my query and have confirmed that I am getting the same numbers you are showing on the “hidden” chart for Load and Start Render. I extended out my query to the 1/1/18 run and have confirmed that Load and Start Render have indeed returned to normal. However, fullyLoaded continues to climb. I tried to “reveal” the fullyLoaded chart in HTTP Archive using the same mechanism you shared but was not successful. Perhaps you could run a quick query to confirm the fullyLoaded result I am showing is correct. If it is that begs the question why would onLoad return to “normal” while fullyLoaded continues to climb. I guess one plausible answer is that a very popular 3rd party (i.e., Facebook) continues to have issues but the calls that are causing a problem are firing AFTER onload. Just a theory.
