Loading Speed across JS Frameworks

housseindjirdeh · September 18, 2018, 11:46pm

I wanted to see if it would be possible to track web page performance across different JS frameworks. I knew that WPT already supported Wappalyzer, but was super glad to stumble on @paulcalvano’s excellent analysis of CPU times grouped by framework by extracting it’s data.

Extending the query that he used previously to now find the median FP, FCP, FMP and TTI for desktop:

SELECT jsframework,
       count(*) freq,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(50)]) firstPaint,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(50)] firstContentfulPaint,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(50)] firstMeaningfulPaint, 
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(50)] timeToInteractive
FROM (
  SELECT  REGEXP_REPLACE(
              JSON_EXTRACT(payload,"$._detected.JavaScript Frameworks"),
              r"([0-9.\"\s]+)", 
              "") jsframework,
        CAST(JSON_EXTRACT(payload, "$['_firstPaint']") as FLOAT64) firstPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstContentfulPaint']") as INT64) firstContentfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstMeaningfulPaint']") as INT64) firstMeaningfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_TimeToInteractive']") as INT64) timeToInteractive
  FROM `httparchive.pages.2018_09_01_desktop`
)
GROUP BY jsframework
ORDER BY freq DESC

56%20PM

Now if we filter for popular frameworks and query against mobile:

SELECT jsFramework,
       count(*) freq,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(50)]) firstPaint,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(50)] firstContentfulPaint,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(50)] firstMeaningfulPaint, 
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(50)] timeToInteractive
FROM (
  SELECT  REGEXP_REPLACE(
              JSON_EXTRACT(payload,"$._detected.JavaScript Frameworks"),
              r"([0-9.\"\s]+)", 
              "") jsFramework,
        CAST(JSON_EXTRACT(payload, "$['_firstPaint']") as FLOAT64) firstPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstContentfulPaint']") as INT64) firstContentfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstMeaningfulPaint']") as INT64) firstMeaningfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_TimeToInteractive']") as INT64) timeToInteractive
  FROM `httparchive.pages.2018_09_01_mobile`
)
WHERE 
 jsFramework='React' OR 
 jsFramework='AngularJS' OR 
 jsFramework='Angular' OR 
 jsFramework='Angular,Zonejs' OR 
 jsFramework='Vuejs' OR 
 jsFramework='Polymer'
GROUP BY jsFramework
ORDER BY freq DESC

02%20PM

Some interesting things here :

• Angular,Zonejs has significantly more entries than Angular so I assume Wappalyzer mostly uses this to categorize Angular 2+ applications?
• Polymer results are a bit shocking. ~27s TTI? I would have assumed significantly better results here.

We can extend the SELECT clause to 50/75/95 percentiles:

SELECT jsFramework,
       count(*) freq,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(50)]) firstPaint50,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(75)]) firstPaint75,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(95)]) firstPaint95,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(50)] firstContentfulPaint50,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(75)] firstContentfulPaint75,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(95)] firstContentfulPaint95,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(50)] firstMeaningfulPaint50,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(75)] firstMeaningfulPaint75,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(95)] firstMeaningfulPaint95,
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(50)] timeToInteractive50,
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(75)] timeToInteractive75,
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(95)] timeToInteractive95
....

13%20PM

housseindjirdeh · September 18, 2018, 11:55pm

I would also love to query against page weight (median total JS bytes) across different JS frameworks but not sure how easy that would be. Would it be possible to cross-reference against httparchive.pages?

rviscomi · September 19, 2018, 12:08am

Nice work @housseindjirdeh! Love to see deeper analysis into JS frameworks.

Haven’t double checked but IIRC the result is a comma-separated list of frameworks, so unless you’re splitting by a comma you’re not getting all of the sites with each framework but rather the sites with only those frameworks. Probably not a huge deal since most sites wouldn’t have more than one framework (or so you’d hope) but that may account for the Zonejs thing.

I would also caution against reading too much into loading speed metrics here. HTTP Archive is a synthetic dataset so the performance data may or may not be representative of real user experiences. THAT SAID, there are exciting possibilities when we join it with the Chrome UX Report, which does have FP, FCP, and FID distributions from real users. See CMS Performance for an example of analysis that joins the datasets.

housseindjirdeh · September 19, 2018, 12:18am

Yep! Definitely agree that we can begin to analyze JS framework results a bit more, hence why I kicked off this topic to start some discussion around it.

Thanks a ton for the feedback, I agree that this really isn’t representative of actual user experience - I’ll fiddle around with CrUX datasets and see if I can land on something a bit more tangible!

paulcalvano · September 19, 2018, 2:52am

Nice work on this! I definitely think there’s a lot more we can learn about the cost of frameworks by digging in deeper.

You can join the httparchive.pages table to the httparchive.summary_pages table to pull in some of the page weights. For example, this query adds the the median JavaScript bytes to your table -

SELECT jsframework,
       count(*) freq,
       ROUND(APPROX_QUANTILES(firstPaint, 100)[SAFE_ORDINAL(50)]) firstPaint,
       APPROX_QUANTILES(firstContentfulPaint, 100)[SAFE_ORDINAL(50)] firstContentfulPaint,
       APPROX_QUANTILES(firstMeaningfulPaint, 100)[SAFE_ORDINAL(50)] firstMeaningfulPaint, 
       APPROX_QUANTILES(timeToInteractive, 100)[SAFE_ORDINAL(50)] timeToInteractive,
       APPROX_QUANTILES(bytesJs, 100)[SAFE_ORDINAL(50)] bytesJs
FROM (
  SELECT  REGEXP_REPLACE(
              JSON_EXTRACT(payload,"$._detected.JavaScript Frameworks"),
              r"([0-9.\"\s]+)", 
              "") jsframework,
        CAST(JSON_EXTRACT(payload, "$['_firstPaint']") as FLOAT64) firstPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstContentfulPaint']") as INT64) firstContentfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_firstMeaningfulPaint']") as INT64) firstMeaningfulPaint,
        CAST(JSON_EXTRACT(payload, "$['_TimeToInteractive']") as INT64) timeToInteractive,
        bytesJs
  FROM `httparchive.pages.2018_08_15_desktop` pages
  INNER JOIN `httparchive.summary_pages.2018_08_15_desktop`  summary_pages
  ON pages.url = summary_pages.url
  )
GROUP BY jsframework
ORDER BY freq DESC

(Note: that above query used the 8/15 dataset, since it seems that the summary_pages table does not exist for 9/1 yet)

tomayac · September 19, 2018, 6:28am

In case anyone is interested in understanding Wappalyzer, it’s all happening in this JSON file, which follows this format.

samdutton · September 19, 2018, 2:51pm

Also occurred to me it would be nice to compare load speed across frameworks for pages with similar total payload size, which might make it more likely that you’re comparing like with like (if that makes sense!)

housseindjirdeh · September 19, 2018, 7:46pm

@paulcalvano that’s amazing, thank you so much. So so excited to see how much deeper we can go here

@tomayac cheers thank you. @developit’s kicked off an issue to improve how they detect (just looking for global variables limits their results to sites that aren’t bundled with Webpack/Rollup)

@samdutton Agreed, someone else mentioned the same earlier today. It definitely adds more validity to the numbers to be able to compare along the same payload size. We can also also create separate buckets of data for different payload sizes (100-200 KB, 200 - 300 KB, etc…) so folks can explore any to see metrics

Topic		Replies	Views
Using Wappalyzer to Analyze CPU Times Across JS Frameworks Analysis	4	7781	April 24, 2018
CPU time breakdown Analysis	3	4488	November 8, 2018
CMS Performance Analysis	22	21164	August 15, 2019
Examining Web Performance Metric Trends	0	54	October 14, 2024
Exploring Relationships Between Performance Metrics in HTTP Archive Data Analysis	4	3975	May 14, 2019

Loading Speed across JS Frameworks

Related topics