# Does BigQuery contain HAR Archive or cookies of crawled webpages? **URL:** \ **Category:** Uncategorized\ **Created:** [June 23, 2020, 7:26am UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968 "2020-06-23T07:26:22Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![\_nurullah](https://avatars.discourse-cdn.com/v4/letter/_/6a8cbe/32.png) [@\_nurullah](https://discuss.httparchive.org/u/_nurullah)\ **Post date:** [June 23, 2020, 7:26am UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/1 "2020-06-23T07:26:22Z")
Hi, Do we have also HAR Archive of crawled websites or stored cookies while crawling? I didn’t find any table or column in BigQuery. Am I missing something or don’t we have such data? Bests, Nurullah ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [July 2, 2020, 11:06pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/3 "2020-07-02T23:06:34Z")
The `requests` dataset contains the request/response HAR payload for each request. For example the JSON path `$.response.headers` may include `Set-Cookie` headers. I’m looking for examples to show, but finding results like `{name: "Set-Cookie", value: "[110 bytes were stripped]"}` which aren’t very useful. I found a [related comment](https://discuss.httparchive.org/t/working-with-csv-dumps/1835/6): > [@Working with CSV dumps](https://discuss.httparchive.org/t/working-with-csv-dumps/1835/6): > > The cookie/auth headers were stripped by Chrome’s netlog capture and were just fixed so those won’t be accurate but everything else should be (though automated parsing of them in bigquery can be a pain). @patmeenan is there still an issue with capturing cookies in Chrome? This would be really detrimental to the Cookies chapter we’re planning for the Web Almanac. ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [July 3, 2020, 1:12am UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/4 "2020-07-03T01:12:23Z")
Cookie data should be there: [https://www.webpagetest.org/result/200703\_E4\_2c49cd0de79b2757c89cbaf7557fc65d/1/details/#step1\_request1](https://www.webpagetest.org/result/200703_E4_2c49cd0de79b2757c89cbaf7557fc65d/1/details/#step1_request1) ```sql Set-Cookie: o=aa59c073dffeb41ea8a5e2e4f838d0fa9838da75; expires=Sat, 03-Jul-2021 01:10:11 GMT; Max-Age=31536000; path=/ ``` There may be some edge cases where the request was only available from netlog but they should be rare and far between. ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [July 3, 2020, 9:39pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/5 "2020-07-03T21:39:32Z")
> [@patmeenan](#): > > There may be some edge cases where the request was only available from netlog but they should be rare and far between. I’m seeing ~30% of cookies having stripped bytes: ```sql CREATE TEMPORARY FUNCTION extractHeader(payload STRING, name STRING) RETURNS STRING LANGUAGE js AS ''' try { var $ = JSON.parse(payload); var header = $._headers.response.find(h => h.toLowerCase().startsWith(name.toLowerCase())); if (!header) { return null; } return header.substr(header.indexOf(':') + 1).trim(); } catch (e) { return null; } '''; SELECT COUNTIF(REGEXP_CONTAINS(cookie, r'bytes were stripped')) AS stripped_bytes, COUNT(0) AS total_reqs, COUNTIF(REGEXP_CONTAINS(cookie, r'bytes were stripped')) / COUNT(0) AS pct_stripped_bytes FROM ( SELECT extractHeader(payload, 'Set-Cookie') AS cookie FROM `httparchive.requests.2020_06_01_mobile`) WHERE cookie IS NOT NULL ``` | stripped\_bytes | total\_reqs | pct\_stripped\_bytes | | --- | --- | --- | | 19,671,309 | 68,839,626 | 28.58% | ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [July 3, 2020, 10:31pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/6 "2020-07-03T22:31:17Z")
Any chance you can provide a few page URLs (and the request URLs) where it is happening? If I can reproduce it I should be able to fix it. ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [July 3, 2020, 10:49pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/7 "2020-07-03T22:49:01Z")
Here’s a random sample of 20 instances: | wptid | cookie | page | url | | --- | --- | --- | --- | | 200512\_Mx3S\_7CYS | [159 bytes were stripped] | [http://www.7-eleven-th.club/](http://www.7-eleven-th.club/) | [https://d.agkn.com/pixel/6639/?che=1589291970&sk=164751303419001301530&pd=&py=&b2b=&al=&cec=&wmt=&l0=https://idsync.rlcdn.com/379128.gif?partner\_uid=164751303419001301530](https://d.agkn.com/pixel/6639/?che=1589291970&sk=164751303419001301530&pd=&py=&b2b=&al=&cec=&wmt=&l0=https://idsync.rlcdn.com/379128.gif?partner_uid=164751303419001301530) | | 200507\_Mx8S\_H6AR | [793 bytes were stripped] | [http://danaghaibasli.over-blog.com/](http://danaghaibasli.over-blog.com/) | [https://pr-bh.ybp.yahoo.com/sync/rubicon/WT6gDWxlaLKDkisnfI6REMn5EUdSAgOZEtemQ7w0kco?csrc=](https://pr-bh.ybp.yahoo.com/sync/rubicon/WT6gDWxlaLKDkisnfI6REMn5EUdSAgOZEtemQ7w0kco?csrc=) | | 200505\_Mx68\_C6WG | [315 bytes were stripped] | [http://rabour.ir/](http://rabour.ir/) | [https://www.instagram.com/p/BR8DpvjBbU0/media/?size=t](https://www.instagram.com/p/BR8DpvjBbU0/media/?size=t) | | 200517\_Mx5A\_AC30 | [138 bytes were stripped] | [https://chto-posmotret.online/](https://chto-posmotret.online/) | [https://sync.1dmp.io/pixel.gif?cid=3cbc2ec8-1421-4677-89fe-2ac6fc52a09a&pid=w&o=au&cs=1](https://sync.1dmp.io/pixel.gif?cid=3cbc2ec8-1421-4677-89fe-2ac6fc52a09a&pid=w&o=au&cs=1) | | 200507\_Mx5F\_ANZ5 | [138 bytes were stripped] | [https://m.tut.by/](https://m.tut.by/) | [https://sync.1dmp.io/pixel.gif?cid=3cbc2ec8-1421-4677-89fe-2ac6fc52a09a&pid=w&o=au&cs=1](https://sync.1dmp.io/pixel.gif?cid=3cbc2ec8-1421-4677-89fe-2ac6fc52a09a&pid=w&o=au&cs=1) | | 200518\_Mx12\_2370 | [167 bytes were stripped] | [https://bloodstained.fandom.com/](https://bloodstained.fandom.com/) | [https://sync.srv.stackadapt.com/sync?nid=1&gdpr=&gdpr\_consent=](https://sync.srv.stackadapt.com/sync?nid=1&gdpr=&gdpr_consent=) | | 200513\_Mx7V\_FB2N | [126 bytes were stripped] | [https://kahramandabidi.blogspot.com/](https://kahramandabidi.blogspot.com/) | [https://dpm.demdex.net/ibs:dpid=121998&dpuuid=77c9f707fdb617a7643eb5c81461b7df&redir=https%3A%2F%2Fsync.crwdcntrl.net%2Fmap%2Fc%3D9828%2Ftp%3DADBE%2Ftpid%3D%24{DD\_UUID}](https://dpm.demdex.net/ibs:dpid=121998&dpuuid=77c9f707fdb617a7643eb5c81461b7df&redir=https%3A%2F%2Fsync.crwdcntrl.net%2Fmap%2Fc%3D9828%2Ftp%3DADBE%2Ftpid%3D%24%7BDD_UUID%7D) | | 200520\_Mx3T\_7EC1 | [160 bytes were stripped] | [https://www.justjaredjr.com/](https://www.justjaredjr.com/) | [https://ssc-cms.33across.com/ps/?ri=102&ru=https%3A%2F%2Fcms-xch-chicago.33across.com%2Fmatch%3Fbidder\_id%3D102%26ttl%3D1592594860%26external\_user\_id%3D77451614-7018-44e3-b4d2-dfd14443fd15](https://ssc-cms.33across.com/ps/?ri=102&ru=https%3A%2F%2Fcms-xch-chicago.33across.com%2Fmatch%3Fbidder_id%3D102%26ttl%3D1592594860%26external_user_id%3D77451614-7018-44e3-b4d2-dfd14443fd15) | | 200512\_Mx7J\_ESBA | [94 bytes were stripped] | [https://podlodka.info/](https://podlodka.info/) | [https://px.adhigh.net/p/cm/sape?u=0100007FB83ABB5E2C02E53D02A36821&bounced=1](https://px.adhigh.net/p/cm/sape?u=0100007FB83ABB5E2C02E53D02A36821&bounced=1) | | 200511\_Mx5E\_AMR0 | [82 bytes were stripped] | [https://lepetitgael.wordpress.com/](https://lepetitgael.wordpress.com/) | [https://x.bidswitch.net/sync?ssp=the33across&custom\_data=&gdpr=0&gdpr\_consent=](https://x.bidswitch.net/sync?ssp=the33across&custom_data=&gdpr=0&gdpr_consent=) | | 200518\_Mx3E\_6RPG | [148 bytes were stripped] | [https://primbon-mimpigigi.blogspot.com/](https://primbon-mimpigigi.blogspot.com/) | [https://loadm.exelator.com/load/?p=204&g=260&buid=4dc5ce95d2d8f1e8d4a23131adaeed54&j=0&xl8blockcheck=1](https://loadm.exelator.com/load/?p=204&g=260&buid=4dc5ce95d2d8f1e8d4a23131adaeed54&j=0&xl8blockcheck=1) | | 200511\_Mx5D\_AK0T | [236 bytes were stripped] | [http://gudangilmu79.blogspot.com/](http://gudangilmu79.blogspot.com/) | [https://id.rlcdn.com/466606.gif?cparams=google\_push%3DAQvitUJPO4KUr69UETlaQWrvD8FPkB3l8iueH9ZT1TKth8EE1WSspjG1FYnXnXFbxa5k4hiuoaqoFtFZ4WDGjSAMty5MLsuRS-yK&google\_gid=CAESEEi11PyUMIlcWnnZE2fVp7w&google\_cver=1](https://id.rlcdn.com/466606.gif?cparams=google_push%3DAQvitUJPO4KUr69UETlaQWrvD8FPkB3l8iueH9ZT1TKth8EE1WSspjG1FYnXnXFbxa5k4hiuoaqoFtFZ4WDGjSAMty5MLsuRS-yK&google_gid=CAESEEi11PyUMIlcWnnZE2fVp7w&google_cver=1) | | 200510\_Mx8C\_GCJS | [143 bytes were stripped] | [https://www.plink.com.co/](https://www.plink.com.co/) | [http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt](http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt) | | 200518\_Mx3Y\_7Q1W | [111 bytes were stripped] | [https://westland-survival.fandom.com/](https://westland-survival.fandom.com/) | [https://nep.advangelists.com/xp/user-sync?acctid=319&redirect=https%3A%2F%2Fcms-xch-chicago.33across.com%2Fmatch%3Fbidder\_id%3D100%26external\_user\_id%3D{PARTNER\_VISITOR\_ID}](https://nep.advangelists.com/xp/user-sync?acctid=319&redirect=https%3A%2F%2Fcms-xch-chicago.33across.com%2Fmatch%3Fbidder_id%3D100%26external_user_id%3D%7BPARTNER_VISITOR_ID%7D) | | 200520\_Mx16\_2AN1 | [138 bytes were stripped] | [https://www.digitaltrends.com/](https://www.digitaltrends.com/) | [https://pixel-eu.rubiconproject.com/exchange/sync.php?p=sovrn-onscroll&gdpr=0&gdpr\_consent=](https://pixel-eu.rubiconproject.com/exchange/sync.php?p=sovrn-onscroll&gdpr=0&gdpr_consent=) | | 200517\_Mx5K\_AY6Z | [315 bytes were stripped] | [http://rumfanatic.pl/](http://rumfanatic.pl/) | [https://www.instagram.com/p/B\_kmYh1Fdt-/media/?size=t](https://www.instagram.com/p/B_kmYh1Fdt-/media/?size=t) | | 200515\_Mx21\_3ZK6 | [68 bytes were stripped] | [https://anikuribon.net/](https://anikuribon.net/) | [https://image6.pubmatic.com/AdServer/UCookieSetPug?oid=1&rd=https%3A%2F%2Fcm.g.doubleclick.net%2Fpixel%3Fgoogle\_nid%3Dpmeb%26google\_sc%3D1%26google\_hm%3D%23%23B64\_16B\_PM\_UID%26google\_redir%3Dhttps%253A%252F%252Fimage8.pubmatic.com%252FAdServer%252FImgSync%253Fsec%253D1%2526p%253D156578%2526mpc%253D4%2526fp%253D1%2526pu%253Dhttps%25253A%25252F%25252Fimage4.pubmatic.com%25252FAdServer%25252FSPug%25253Fp%25253D156578%252526sc%25253D1&google\_gid=CAESECp3S7jyaoyvP7ctUJc0xZQ&google\_cver=1&google\_push=AQvitUKXYkH15dM94gEbd7TC4memHTeWNCCWkUTjiDkzdk-tSHyiThxMqcXgZO5bQRxH8vvAUG63d96CGlaiT3Qy4tJSa8bbxjEZOQ](https://image6.pubmatic.com/AdServer/UCookieSetPug?oid=1&rd=https%3A%2F%2Fcm.g.doubleclick.net%2Fpixel%3Fgoogle_nid%3Dpmeb%26google_sc%3D1%26google_hm%3D%23%23B64_16B_PM_UID%26google_redir%3Dhttps%25253A%25252F%25252Fimage8.pubmatic.com%25252FAdServer%25252FImgSync%25253Fsec%25253D1%252526p%25253D156578%252526mpc%25253D4%252526fp%25253D1%252526pu%25253Dhttps%2525253A%2525252F%2525252Fimage4.pubmatic.com%2525252FAdServer%2525252FSPug%2525253Fp%2525253D156578%25252526sc%2525253D1&google_gid=CAESECp3S7jyaoyvP7ctUJc0xZQ&google_cver=1&google_push=AQvitUKXYkH15dM94gEbd7TC4memHTeWNCCWkUTjiDkzdk-tSHyiThxMqcXgZO5bQRxH8vvAUG63d96CGlaiT3Qy4tJSa8bbxjEZOQ) | | 200517\_Mx26\_48T3 | [131 bytes were stripped] | [https://voala.org/](https://voala.org/) | [https://match.adsrvr.org/track/cmf/generic?ttd\_pid=tapad&ttd\_tpi=1&ttd\_puid=b7c87dc1-9835-11ea-8792-928c11c458fb%2C&gdpr=0&gdpr\_consent=](https://match.adsrvr.org/track/cmf/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=b7c87dc1-9835-11ea-8792-928c11c458fb%252C&gdpr=0&gdpr_consent=) | | 200502\_MxF\_Y1S | [353 bytes were stripped] | [https://grit.trixstar.com/](https://grit.trixstar.com/) | [https://d.adroll.com/cm/n/out?adroll\_fpc=df4d7f8d468d8e6e6e633ef313240586-1588396158332&arrfrr=https%3A%2F%2Fgrit.trixstar.com%2F&advertisable=JTND4B4H4RDJNAW722HN74](https://d.adroll.com/cm/n/out?adroll_fpc=df4d7f8d468d8e6e6e633ef313240586-1588396158332&arrfrr=https%3A%2F%2Fgrit.trixstar.com%2F&advertisable=JTND4B4H4RDJNAW722HN74) | | 200510\_Mx84\_FWG2 | [373 bytes were stripped] | [https://furniture-ideal.com/](https://furniture-ideal.com/) | [https://www.linkedin.com/px/li\_sync?redirect=https%3A%2F%2Fpx.ads.linkedin.com%2Fcollect%3Fv%3D2%26fmt%3Djs%26pid%3D244251%26url%3Dhttps%3A%2F%2Ffurniture-ideal.com%2F%26time%3D1589160084639%26cookiesTest%3Dtrue%26liSync%3Dtrue](https://www.linkedin.com/px/li_sync?redirect=https%3A%2F%2Fpx.ads.linkedin.com%2Fcollect%3Fv%3D2%26fmt%3Djs%26pid%3D244251%26url%3Dhttps%253A%252F%252Ffurniture-ideal.com%252F%26time%3D1589160084639%26cookiesTest%3Dtrue%26liSync%3Dtrue) | ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [July 3, 2020, 11:18pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/8 "2020-07-03T23:18:44Z")
Looks like it might be redirects that drop cookies that don’t get matched up. I’ll take a look over the weekend and see if I can track it down. ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [July 3, 2020, 11:20pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/9 "2020-07-03T23:20:03Z")
Thanks Pat! No need to work over the holiday weekend, it can wait until next week 🙂 ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [July 5, 2020, 12:25am UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/10 "2020-07-05T00:25:36Z")
Think I figured it out (and yeah, it’s redirect-specific). The dev tools messages with the real response headers don’t have timestamps and redirects re-use the same request ID. Usually the agent takes care of this automatically by generating artificial ID’s for each step of the redirect path by keeping track of that the current ID is for a given dev-tools request ID. The agent also sorts the events by timestamp which causes a problem for the header events that don’t have a timestamp so they don’t get the automatic request ID remapping. The dev tools events are in the correct order naturally so I got rid of the sort and it worked in local testing. Waiting for the release to roll out over the next hour to make sure it works correctly. That means most of the July crawl will have correct headers for the redirect case but it won’t be until August where it’s 100% correct. ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [July 5, 2020, 12:47am UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/11 "2020-07-05T00:47:41Z")
Yep, that appears to have done it: [https://www.webpagetest.org/result/200705\_XS\_0e34853480e95f3ef066c06d9e49791b/1/details/#step1\_request45](https://www.webpagetest.org/result/200705_XS_0e34853480e95f3ef066c06d9e49791b/1/details/#step1_request45) Hopefully there aren’t any unintended side-effects. ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [July 5, 2020, 4:59pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/12 "2020-07-05T16:59:13Z")
Thanks for fixing @patmeenan 🎉 ---
**Author:** ![ydimova](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/ydimova/32/1891_2.png) [@ydimova](https://discuss.httparchive.org/u/ydimova)\ **Post date:** [June 11, 2021, 2:55pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/13 "2021-06-11T14:55:18Z")
Could it be that issue is still present @rviscomi @patmeenan ? In the latest requests desktop table, some cookie value are stripped. We were hoping to analyze cookies in the privacy chapter of the Web Almanac 2021. ---
**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\ **Post date:** [June 11, 2021, 5:49pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/14 "2021-06-11T17:49:12Z")
I reran the query from [earlier in the thread](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/5) against May 2021 data and the results show only 1.5% of cookies with stripped bytes. So you’re right that there’s still some data loss but it’s a much better situation than last year. @patmeenan is this about what you’d expect or is it feasible to get down to 0%? ---
**Author:** ![patmeenan](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/patmeenan/32/605_2.png) [@patmeenan](https://discuss.httparchive.org/u/patmeenan)\ **Post date:** [June 11, 2021, 6:17pm UTC](https://discuss.httparchive.org/t/does-bigquery-contain-har-archive-or-cookies-of-crawled-webpages/1968/15 "2021-06-11T18:17:32Z")
I imagine that resolving the remaining bit will probably require changes to Chrome (and a fair bit of investigation to figure out what is going on). Anything is possible but I wouldn’t hold out for it soon.