I’m writing a research paper and need to explain how the http archive crawler declares itself to websites as a bot. can you please explain this to me? I’m particularly interested in learning if it’s included on this list: IAB Tech Lab
Thank you
Sorry I missed this earlier. We identify ourselves with the PTST (for WebPageTest) in the user agent:
Mozilla/5.0 (Linux; Android 8.1.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Mobile Safari/537.36 PTST/251102.222103
I don’t know if the IAB Tech Lab includes us as their list is not publicly available. Would be interested to hear if anyone with access has details on this.