How are reqTotal, reqImg, bytesTotal and bytesImg calculated?

1

Need help on understanding the 4 page level attributes collected.

  1. How is reqTotal calculated? How is reqImg calculated?
    Is reqTotal equal to sum of the following?
    reqHtml, reqJS, reqCSS, reqGif, reqJpg, reqPng, reqFont, reqFlash, reqJson, reqOther
    As an example, for Aug 1, 2017, http://www.google.com/,
    reqTotal is 15 and reqImg is 6 but reqHtml(4), reqJS(4), reqPng(5) and reqFont(2) do not add up.

  2. Similarly bytesTotal does not appear to match the sum of constituents?
    bytesTotal is 415905, but
    bytesHtml(70133), bytesJS(279573), bytes Img(36919), bytesPng(35425) and bytesFont(29280)

  3. Is bytes the amount of data that is actually transferred on the wire (in compressed form)?

3

Hi @raghuramakrishnan. Good questions. Here are my thoughts on them.

Requests

  • reqTotal is the sum of all requests to fully load the page.
  • reqImg is the sum of all requests where type=image. This can be png, jpeg, gif, etc.
  • The reqGif, reqJpg, reqPng columns are all subsets of reqImg.
  • When I look at the WPT measurement that was used to generate the data for www.google.com on Aug 1 (http://httparchive.webpagetest.org/result/170801_1_5QW/), I can see 16 requests (essentially the sum of reqHtml+reqJS+reqCSS+reqImg+reqFont)
SELECT pageid, label, url, wptrun, reqTotal, reqHtml, reqJS, reqCSS, reqImg, reqFont, reqFlash, reqJson, reqOther
FROM httparchive.runs.2017_08_01_pages
WHERE wptid = "170801_1_5QW"

image

In the requests table, you can look at this via the type and format columns. type may be image, but format will contain the image format.

SELECT type, format, COUNT(*) freq
FROM `httparchive.runs.2017_08_01_requests` 
WHERE pageid = 79607379
GROUP BY type, format

In this case you can see that looking at reqPng (type=image and format=png) instead of reqImg (type=image) will result in being off by 1 request:
image

Bytes

  • In your example, you are summing the bytesImg and bytesPng which will overcount the number of bytes.
  • As for bytesTotal - you are correct. This represents the amount of data that is actually transferred over the wire (including compressed content)

You can see the correlation of data in the pages table with the data actually recorded by WebPageTest below:

SELECT * 
FROM httparchive.runs.2017_08_01_pages
WHERE wptid = "170801_1_5QW"

image

http://httparchive.webpagetest.org/result/170801_1_5QW/1/breakdown/

image

Hope that helps!

1 Like
4

Hi @paulcalvano,

Thanks for the very useful information. It helps.

a) Is there a difference between bytesHtml and bytesHtmlDoc. Looking at the individual requests, the mime type for all the 4 html requests are text/html.
b) If we look at the breakup in the pie chart and the row in the table, the bytesHtmlDoc (69,902) is a match. However, bytesHtml (70,133) appears to be left out in the totalBytes and pie chart (or may be my interpretation is incorrect)

totalbytes (415,905) = bytesHtmlDoc (69,902) + bytesImage (36,919) + bytesFont (29,280) + bytesJs (279,573) = 415,674

5

The bytesHtml metric is the total size of all of the text/html responses on the page. bytesHtmlDoc is the total size of just the main HTML document. And bytesHtmlDoc is 69,902 bytes, which is the number of bytes downloaded for https://www.google.com/?gws_rd=ssl (the second request)

2 of the other text/html responses returned 0 bytes (there were 204 responses).

The initial request for http://www.google.com/, returned a HTTP 302 redirect to the secure page. That redirect contains 231 bytes HTML. And if you add 69,902 + 231, then you get the 70,133 bytes from bytesHtml.

image

However, that raises another question. Should that pie chart contain the value from bytesHtml or bytesHtmlDoc? Considering it is the content breakdown for the entire page, I would think it should be a summary of all of the text/html resources and not just the main document. @pmeenan- is this a bug in the WPT content breakdown visualization?