# Quickstart guide to exploring the HTTP Archive **URL:** \ **Category:** FAQ\ **Created:** [March 1, 2016, 1:12am UTC](https://discuss.httparchive.org/t/quickstart-guide-to-exploring-the-http-archive/682 "2016-03-01T01:12:43Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![igrigorik](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/igrigorik/32/925_2.png) [@igrigorik](https://discuss.httparchive.org/u/igrigorik)\ **Post date:** [March 1, 2016, 1:12am UTC](https://discuss.httparchive.org/t/quickstart-guide-to-exploring-the-http-archive/682/1 "2016-03-01T01:12:43Z")
**Curious about how the web is built and is evolving over time?** You’re in the right place. Every month, HTTP Archive crawls the top sites on the web and records everything it sees: request and response headers for each and every request on each page, timing and profiling data, and even the response bodies for the text-based resources–to learn more about this process, check out [the FAQ](http://httparchive.org/about.php). This data is what powers the trends and insights you find on this site and it is also available for you to query and explore. Ready to get started? Let’s explore the available options for how to analyze this data… ### Visit HTTP Archive stats and trends As a first step, check out the precomputed summary reports in your browser: - [Stats report](http://httparchive.org/interesting.php) provides a per-crawl report on how the web-pages are constructed - e.g. their size, number of images, use of web fonts, and so on. - [Trends report](http://httparchive.org/trends.php) provides a time-series view on how the key aspects of the webpages—the ones highlighted in the stats reports—are changing over time. The data exposed in above reports is not an exhaustive list of available metrics. In fact, there is _a lot_ more to explore and if you can’t find what you’re looking for, or want to dig deeper into particular trend or metric, then you can do so via one of the following routes. ### Explore the HTTP Archive dataset HTTP Archive is powered by WebPageTest. Each page is crawled using the WPT agent and the results, which include request and response headers, text response bodies, and profiling data are archived. This yields ~1TB of data for each and every crawl. There are two options to access this data for your own analysis: 1. [Download it](https://discuss.httparchive.org/t/how-to-download-the-http-archive-data/679) for local analysis. 2. Use one of the cloud tools below to explore the data. The former allows you to use own and local tools to explore the data, but comes with a high upfront download and setup costs. The latter option provides easy and instant “cloud access” to the data, but requires a Google developer account and, depending on the amount of type of analysis you run, may have extra costs. #### Exploring via BigQuery All of the HTTP Archive crawl data is available on Google BigQuery, which is a warehouse and an engine for large-scale data analytics: - You can [write and run ~SQL queries from the comfort of your browser](https://cloud.google.com/bigquery/web-ui-quickstart) (or [CLI](https://cloud.google.com/bigquery/bq-command-line-tool)) - It has built in [URL, statistical, windowing, and other useful functions](https://cloud.google.com/bigquery/query-reference) - The first 1 TB of data processed per month is [free of charge](https://cloud.google.com/bigquery/pricing#free-tier) There are two datasets that you can query: 1. **[https://bigquery.cloud.google.com/dataset/httparchive:har](https://bigquery.cloud.google.com/dataset/httparchive:har)** - these tables contain the full denormalized HAR output for each and every page in the crawl. The navigation requests are stored in the `pages` table, and subresource requests are recorded as separate rows in the corresponding `requests` table. - Take a look at the [pages](https://bigquery.cloud.google.com/table/httparchive:har.chrome_jan_1_2016_pages) and [requests](https://bigquery.cloud.google.com/table/httparchive:har.chrome_jan_1_2016_requests) table schemas. Tip: click “preview” to see example data. - Note that this dataset keeps most of its data inside the `payload` fields that contain JSON data. This is intentional, to allow the schema to change with time. To access fields inside of these payloads, use the [JSON functions provided by BigQuery](https://cloud.google.com/bigquery/query-reference#jsonfunctions). 1. **[https://bigquery.cloud.google.com/dataset/httparchive:runs](https://bigquery.cloud.google.com/dataset/httparchive:runs)** - these tables contain summary reports for each crawl, with a collection of precomputed metrics. - Take a look at the [pages](https://bigquery.cloud.google.com/table/httparchive:runs.2016_01_01_pages) and [requests](https://bigquery.cloud.google.com/table/httparchive:runs.2016_01_01_requests) schemas. For some fun and insightful examples of using BigQuery, check out one of many [hands-on examples in our Analysis section](https://discuss.httparchive.org/c/analysis/l/top), and the following lightning talk from Velocity conference: #### Exploring with Datalab If you want to go beyond running a simple query and compose a report or do more in-depth analysis, consider using the [Cloud Datalab](https://cloud.google.com/datalab/): you can query BigQuery from the Jupyter notebook, crunch and visualize the results with Pandas, matplotlib, and other popular tools. Demo: [https://github.com/HTTPArchive/bigquery/blob/master/datalab/histograms.ipynb](https://github.com/HTTPArchive/bigquery/blob/master/datalab/histograms.ipynb) #### Crunching data with Dataflow The most involved—and the most powerful—way to analyze the full HTTP Archive dataset is via [Dataflow](https://cloud.google.com/dataflow/), which enables you to write and run custom ETL and batch computation jobs in the cloud. For example, HTTP Archive uses dataflow to load the raw [HAR archives](http://www.softwareishard.com/blog/har-12-spec/) provided by the WebPageTest into BigQuery. Demo: [https://github.com/HTTPArchive/bigquery/tree/master/dataflow](https://github.com/HTTPArchive/bigquery/tree/master/dataflow)