# No JSON requests 2022-2024 **URL:** \ **Category:** Analysis\ **Created:** [March 25, 2026, 11:07am UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003 "2026-03-25T11:07:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Wuteken](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@Wuteken](https://discuss.httparchive.org/u/Wuteken)\ **Post date:** [March 25, 2026, 11:07am UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/1 "2026-03-25T11:07:20Z")
I’m investigating the evolution of the web over time for my masters thesis. One of the things I wanted to map out was the percentage of every type of request per year. When constructing this graph I realized there were exactly 0 requests with type ‘json’ found in the whole dataset during the period 2022-2024. I specifically tested this for the dates 2022-06-01, 2023-01-01, 2023-06-01, 2024-01-01, 2024-02-01, 2024-06-01 using the query below: ```sql SELECT COUNT(1) FROM `httparchive.crawl.requests` where type = 'json' AND date = '2024-02-01' ``` I have 2 questions regarding this: 1. Is there a specific reason for the missing json requests in these years? 2. Are there any other requests/request-types missing that I should know about for my thesis? Thank you in advance for your answer. ---
**Author:** ![tunetheweb](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/tunetheweb/32/1671_2.png) [@tunetheweb](https://discuss.httparchive.org/u/tunetheweb)\ **Post date:** [March 25, 2026, 11:36am UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/2 "2026-03-25T11:36:54Z")
Before the November 2024 crawl which includes [this change](https://github.com/HTTPArchive/wptagent/pull/20/changes), JSON didn’t have it’s own type and was treated as a `script` type. As you’ve not doubt noticed, querying for as specific type is a lot quicker (and easier!) than looking across all requests, which is why we added the `json` type in the first place. We’re continually enhancing the dataset as we find more improvements such as this, but don’t always backport the fixes since it may cause confusion to those rerunning queries. We don’t explicitly list every single change in a changelog (though the major ones are [listed here](https://httparchive.org/faq#what-changes-have-been-made-to-the-test-environment-that-might-affect-the-data)), so it’s really a matter of looking at the git history when you spot an issue like this. [This file’s history](https://github.com/HTTPArchive/wptagent/commits/main/HTTPArchive/utils.py) is probably of the most interest and you can see I also improved our media detection in some changes there. ---
**Author:** ![Wuteken](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@Wuteken](https://discuss.httparchive.org/u/Wuteken)\ **Post date:** [March 25, 2026, 12:17pm UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/3 "2026-03-25T12:17:28Z")
Hi, Thank you for your quick response. That file’s history is indeed useful to view for changes after 2024. Where was this code located before 2024? In my data I see that JSON had its own type from 2016-01-01 until 2022-01-01. As you mentioned this made it quicker and easier to query so I was wondering why it was removed. ---
**Author:** ![tunetheweb](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/tunetheweb/32/1671_2.png) [@tunetheweb](https://discuss.httparchive.org/u/tunetheweb)\ **Post date:** [March 25, 2026, 12:46pm UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/4 "2026-03-25T12:46:09Z")
Oh that’s interesting! The data pipeline has been through several iterations - many before my time. This included [a Java one](https://github.com/HTTPArchive/bigquery/tree/master/dataflow/java), which was then [migrated to Python](https://github.com/HTTPArchive/bigquery/tree/master/dataflow/python), which in 2022 was migrated to [a new code base](https://github.com/HTTPArchive/data-pipeline), before in 2024 moving to the current [agent-based one](https://github.com/HTTPArchive/wptagent/tree/main/HTTPArchive). Mostly of these changes were done for reasons of scale as the dataset grew and grew and struggled to cope with the older technologies. So it’s entirely possible we missed something during one of those migrations and so lost some data. Having a quick look for JSON I don’t see any specific logic for that. However, we also migrated data model in that time and I think `type` may have been a new field added in 2022. So it’s also possible when we migrated all the pre-2022 data to the new data model we handled this better as that backfill was done after the previously mentioned change in 2024 IIRC, so basically used that new 2024 logic to backfill. ---
**Author:** ![javiermarinet](https://avatars.discourse-cdn.com/v4/letter/j/9dc877/32.png) [@javiermarinet](https://discuss.httparchive.org/u/javiermarinet)\ **Post date:** [March 25, 2026, 2:38pm UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/5 "2026-03-25T14:38:39Z")
Exactamente, la evolución, no cree? ---
**Author:** ![josetteleonard64](https://avatars.discourse-cdn.com/v4/letter/j/8491ac/32.png) [@josetteleonard64](https://discuss.httparchive.org/u/josetteleonard64)\ **Post date:** [July 26, 2026, 3:43pm UTC](https://discuss.httparchive.org/t/no-json-requests-2022-2024/3003/6 "2026-07-26T15:43:19Z")
if json existed as its own type through early 2022 then suddenly vanished, the requests probably got reclassified rather than actually deleted. might be worth checking the raw content-type headers during that gap—if you’re seeing application/json entries but the type field switched to something else, thats likely what happened.