Best way to extract gtm.js response bodies at scale, BigQuery cost vs raw HAR access?

1

I’m doing research that requires collecting the response bodies of gtm.js (Google Tag Manager) resources across crawled sites, ideally over multiple monthly crawls.

My filter is essentially:

SELECT date, page, url, response_body
FROM `httparchive.crawl.requests`
WHERE client = 'desktop'
  AND is_root_page = TRUE
  AND type = 'script'
  AND CONTAINS_SUBSTR(url, 'gtm.js')
  AND response_body IS NOT NULL

The challenge is cost. Because url isn’t a clustering column, the gtm.js filter can’t prune, so extracting bodies means scanning the (very large) response_body column across each crawl. Over a wide date range this becomes expensive.

I’ve been trying to find a cheaper route and have a couple of questions:

  1. Is there a recommended pattern for extracting a narrow slice of response_body across many crawls cost-effectively? (e.g. clustering tips I might be missing, a sampled/summary table, or a materialization approach the community uses.)

  2. Raw HAR access: I understand the raw crawl data lives in gs://httparchive (crawls/ and crawls_manifest/). I tried reading it with a billing project attached, but the bucket returns 403 on both objects.list and objects.get. Is there a process to request read access to the raw crawl data for? If so, what’s the right way to apply, and are there expectations around cost (requester-pays) or usage?

Thanks very much for maintaining such a valuable dataset, any guidance on the most cost-appropriate approach would be really appreciated.