# How to best filter out adult sites

**URL:** <https://discuss.httparchive.org/t/how-to-best-filter-out-adult-sites/1928>\
**Category:** Analysis\
**Created:** [April 23, 2020, 6:58pm UTC](https://discuss.httparchive.org/t/how-to-best-filter-out-adult-sites/1928 "2020-04-23T18:58:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrisbolin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/chrisbolin/32/1756_2.png) [@chrisbolin](https://discuss.httparchive.org/u/chrisbolin)\
**Post date:** [April 23, 2020, 6:58pm UTC](https://discuss.httparchive.org/t/how-to-best-filter-out-adult-sites/1928/1 "2020-04-23T18:58:49Z")

</div>

I’d like to remove obvious adult sites from my analysis. Searching these forums for “adult” doesn’t turn up anything helpful, and the `_adult_site` column in `summary_pages` is _always_ false (e.g. in `summary_pages.2020_03_01_mobile`).

My best approach so far has just been to remove records with `REGEXP_CONTAINS(url, r"porn|xxx|adult")`. Is there a better way? Maybe a table I’m missing?

---

<div class="post-metadata">

**Author:** ![rviscomi](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/rviscomi/32/835_2.png) [@rviscomi](https://discuss.httparchive.org/u/rviscomi)\
**Post date:** [April 23, 2020, 8:41pm UTC](https://discuss.httparchive.org/t/how-to-best-filter-out-adult-sites/1928/2 "2020-04-23T20:41:13Z")

</div>

Good question. We don’t have a way to detect adult sites currently, the `_adult_site` flag has been deprecated for a while.

Our philosophy has been that adult sites are just another kind of content on the web, so we don’t try to omit them from our routine analyses. In the spirit of better understanding the web, though, it might still be interesting to slice by adult websites and see if there’s any difference in the way they’re built/experienced, so I could see the argument for better detection.

Generally, when we want to improve our detection methodology, we try to bake that in upstream in the [Wappalyzer](https://www.wappalyzer.com/) project. I wonder if they’d be open to taking this on as a new category of websites?

---

<div class="post-metadata">

**Author:** ![chrisbolin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.httparchive.org/chrisbolin/32/1756_2.png) [@chrisbolin](https://discuss.httparchive.org/u/chrisbolin)\
**Post date:** [April 25, 2020, 5:49pm UTC](https://discuss.httparchive.org/t/how-to-best-filter-out-adult-sites/1928/3 "2020-04-25T17:49:10Z")

</div>

> [@rviscomi](#):
>
> we try to bake that in upstream in the [Wappalyzer](https://www.wappalyzer.com/) project

Thanks for the Wappalyzer callout. I’ll check that out!