Technologies Table

[rviscomi]

Hey Greg! Good questions.

Here are some links for more context behind the dataset:

Integrate Wappalyzer platform detection

write wappalyzer detections to dedicated `technologies` table

Really big queries on BigQuery

The HTTP Archive dataset is over 200 TB all together. And it’s growing at a rate of about 5 TB per month. Sometimes, you need to query over a huge part of it. For example, I’ve been running queries to apply the Wappalyzer technology detection logic over all of the HTML response bodies going back to January 2016 when we first started collecting them. This detection will allow us to retroactively track the adoption of technologies like WordPress, the entire e-commerce category, etc. As you can im…

---

To your questions:

@gregorywolf

Can a site’s framework info be trusted?

The data is only as good as its detection. We use Wappalyzer to do the detection in WebPageTest. And while we’re reasonably confident in the detection, there may be some blind spots, for example frameworks that hide their presence from the global scope. @developit has made some efforts to improve this detection upstream in Wappalyzer.

I think the data is solid enough to do an analysis of framework performance (eg see CMS Performance - #16 by rviscomi for similar analysis).

@gregorywolf

For example I am curious about amazon.com and cannot find any data recorded for Amazon.

Are you looking up the right URL? For example this query shows 78 Amazon URLs in the most recent technologies table:

SELECT
  DISTINCT url
FROM
  `httparchive.technologies.2019_02_01_desktop`
WHERE
  NET.REG_DOMAIN(url) = 'amazon.com'

and there are some entries for https://www.amazon.com/ specifically (AWS and Cloudfront).