Hi @Tim, we’ve already chatted over DM but I just wanted to reiterate a few things here for anyone else who finds this thread.
First, I’m really sorry this happened to you. I can imagine how horrifying it would be to get an unexpected bill for that amount. At your suggestion, I’ve created a PR to add a more explicit warning about this to the website’s FAQ page, which I think is where you discovered the BigQuery dataset, so everyone else will know about the risks before diving in.
99% of the people who access HTTP Archive’s data do so through the free monthly reports and annual Web Almanac reports. BigQuery is really only for the 1% of power users who need lower-level access to the raw data.
In your case, $14,000 works out to processing about 2.5 petabytes worth of data. I don’t know exactly what query you wrote to do that much work, but here’s an example of a query that incurs a similar cost:
Behind the friendly-looking checkmark is a very very important warning: “This query will process 2.66 PB when run.” At $6.25 per TiB that’s about the cost you ran into. I won’t get into the dangers of SELECT * but you get the idea.
Again, I’m very sorry about this and I hope it doesn’t happen to anyone else. Also, as we discussed, I’ll see if there’s any way I can escalate your support ticket internally.
I hope you’re able to set up new cost controls to regain your trust in using BigQuery and continue using the HTTP Archive dataset without any more surprises. And for anyone else looking to get started with the dataset, know that you do not need to provide a credit card to use BigQuery. This will automatically put you in the free tier, which gives you a 1 TB/month quota, and you’ll never be charged a cent. For tips on getting the most out of the dataset for 1 TB/month, check out our guide on minimizing query costs.