# Analyzing traces

**URL:** <https://discuss.httparchive.org/t/analyzing-traces/1655>\
**Category:** Analysis\
**Created:** [May 21, 2019, 5:02pm UTC](https://discuss.httparchive.org/t/analyzing-traces/1655 "2019-05-21T17:02:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [May 21, 2019, 5:02pm UTC](https://discuss.httparchive.org/t/analyzing-traces/1655/1 "2019-05-21T17:02:27Z")

</div>

HTTP Archive collects millions of traces each month. These contain diagnostic information about what Chrome is doing under the hood. See [about:tracing](https://www.chromium.org/developers/how-tos/trace-event-profiling-tool) for more info.

For example, for each test ID we have a corresponding trace file available for download at `http://httparchive.webpagetest.org/getgzip.php?test={TEST_ID}&compressed=1&file=1_trace.json.gz`. The test IDs correspond to the `wptid` field in the [summary\_pages](https://bigquery.cloud.google.com/dataset/httparchive:summary_pages) tables.

In order to do bulk analysis of the traces, we’d need to map the rows in the latest summary\_pages table to download URLs, download and unzip each trace, and pass them through a tool than can do the analysis.

Here’s a query that can do the ID mapping:

```sql
SELECT
  url,
  CONCAT(
    'http://httparchive.webpagetest.org/getgzip.php?test=',
    wptid,
    '&compressed=1&file=1_trace.json.gz'
  ) AS trace_url
FROM
  `httparchive.summary_pages.2019_04_01_mobile`

```

Alternatively, @patmeenan are these traces available somewhere that can be more easily transferred over the network, like GCS or FTP? Probably don’t want 5 million download requests hitting the server.

cc @slightlylate, who has shown an interest in doing this kind of bulk analysis

---

<div class="post-metadata">

**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\
**Post date:** [May 21, 2019, 5:37pm UTC](https://discuss.httparchive.org/t/analyzing-traces/1655/2 "2019-05-21T17:37:18Z")

</div>

Yikes, please fetch them from GCS if possible, they are all in the “httparchive” bucket. That will prevent having to make round trips to the server.

They are stored with all of the traces for a given crawl in a directory for that crawl.

i.e…

```sql
gs://httparchive/traces-android-May_1_2019/
gs://httparchive/traces-chrome-May_1_2019/

```

The files themselves are .json.gz:

```sql
gs://httparchive/traces-android-May_1_2019/190501_Mx10_204X.json.gz

```

If you prefer, you can fetch them over HTTPS:

```sql
https://storage.googleapis.com/httparchive/traces-android-May_1_2019/190501_Mx10_204X.json.gz

```

---

<div class="post-metadata">

**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\
**Post date:** [May 21, 2019, 5:38pm UTC](https://discuss.httparchive.org/t/analyzing-traces/1655/3 "2019-05-21T17:38:53Z")

</div>

btw, I’m not sure where the documentation for it is right now but there was a Chromium map/reduce tool for processing traces in bulk a few years ago that could process the trace files directly from the HTTP Archive cloud storage bucket.