# Data collection in HTTPArchive

**URL:** <https://discuss.httparchive.org/t/data-collection-in-httparchive/1549>\
**Category:** Analysis\
**Created:** [January 15, 2019, 2:15am UTC](https://discuss.httparchive.org/t/data-collection-in-httparchive/1549 "2019-01-15T02:15:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nayanamanas](https://avatars.discourse-cdn.com/v4/letter/n/c67d28/32.png) [@nayanamanas](https://discuss.httparchive.org/u/nayanamanas)\
**Post date:** [January 15, 2019, 2:15am UTC](https://discuss.httparchive.org/t/data-collection-in-httparchive/1549/1 "2019-01-15T02:15:50Z")

</div>

We extracted URLs from harachrive.runs\* table (as in [Adoption of HTTP Security Headers on the Web](https://discuss.httparchive.org/t/adoption-of-http-security-headers-on-the-web/1259)) to study security headers in HTTPArchive data. During this exercise, we noted the number of URLs for 5 specific days (with a gap of few months between any 2 days) does not follow a particular trend (e.g., with time the number URLs in harachrive.runs\* table are not always increasing). As shown below, the number of URLs dropped on 12-01-18 (1262020), compared to previous data dump on 01-08-18 (1277461).

| date | URLs |
| --- | --- |
| 2017\_07\_01 | 475122 |
| 2018\_01\_01 | 490241 |
| 2018\_05\_01 | 504672 |
| 2018\_08\_01 | 1277461 |
| 2018\_12\_01 | 1262020 |

Therefore, we would like to know the follows:

- Isn’t the number of URLs used (in harachrive.runs\* table) increase with time? If not why?
- What is the procedure used to extract headers? ( e.g., follow all redirection and consider the headers in the final landing page if multiple redirection are involved in a page load)

Thanks  
Naya

---

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [January 15, 2019, 6:19pm UTC](https://discuss.httparchive.org/t/data-collection-in-httparchive/1549/2 "2019-01-15T18:19:58Z")

</div>

See [Changes to the HTTP Archive corpus](https://discuss.httparchive.org/t/changes-to-the-http-archive-corpus/1539) for more info on when and why the number of URLs changed.

There is a `firstHtml` boolean field in the `summary_requests` dataset that indicates whether the request returns the root document of the page, after redirects. You can use it like this:

```sql
SELECT
  respOtherHeaders
FROM
  `httparchive.latest.summary_requests_desktop`
WHERE
  firstHtml

```

Or if you need access to the raw JSON request object, you can join the tables together:

```sql
SELECT
  page,
  url,
  JSON_EXTRACT(payload, '$._headers.response') AS all_response_headers
FROM
  `httparchive.latest.summary_requests_desktop`
JOIN
  `httparchive.latest.requests_desktop`
USING
  (url)
WHERE
  firstHtml

```