---
updatedAt: 2025-08-28T17:22:13.000Z
agentTools:
  projectIndex: https://docs.scope3.com/llms.txt
---

# Scope3 Crawler

Scope3's crawler functionality and its purpose for content classification and brand safety assessment.

**Purpose**

The Scope3 crawler indexes publicly available web content and paywalled media (with permission) to assess content classification and brand safety.  It is operated by Scope3 ([scope3.com](http://scope3.com)).

**Enable Scope3 Crawler**

To enable the Scope3 Crawler to scan your content, add it to your approved crawlers list in either your robots.txt file or IP allowlist. Here are the necessary details:

* **User-Agent**: `Scope3/2.0 (scope3.com)`
* **IP Range**: `34.70.81.55`
* **Domain name**: `crawler.scope3.com`

If you have content behind a paywall or subscriber login, please contact us (<customersupport@scope3.com>) to arrange credentials and access for our crawler.

**Crawling Behaviour**

To minimize server impact, we:

* Implement adaptive rate limiting to prevent server overload, with a current limit of 5 pages per minute per domain
* Adjust crawl speeds when encountering server errors (4xx, 5xx)
* You can find more information on our content classification [here](https://docs.scope3.com/docs/content-classification)

**Value for Website Owners**

Website owners can benefit from Scope3's content classification pipeline in the following ways:

* Verify domain ownership through a free Scope3 publisher account at [scope3.com](http://scope3.com), gaining complete visibility into brand safety and vertical model classifications.
* Implement brand safety targeting with our integration tools. Publishers can use our [integration product](https://docs.scope3.com/docs/publisher-real-time-api) to remove the need for crawling.

**Monitoring & Support**

For questions about crawl frequency or suspicious traffic, contact us at <customersupport@scope3.com>. We're committed to addressing your concerns and finding suitable solutions.