Scope3 Crawler

Purpose

The Scope3 crawler indexes publicly available web content and paywalled media (with permission) to assess content classification and brand safety. It is operated by Scope3 (scope3.com).

Enable Scope3 Crawler

To enable the Scope3 Crawler to scan your content, add it to your approved crawlers list in either your robots.txt file or IP allowlist. Here are the necessary details:

• User-Agent: Scope3/2.0 (scope3.com)
• IP Range: 34.70.81.55
• Domain name: crawler.scope3.com

If you have content behind a paywall or subscriber login, please contact us ([email protected]) to arrange credentials and access for our crawler.

Crawling Behaviour

To minimize server impact, we:

• Implement adaptive rate limiting to prevent server overload, with a current limit of 5 pages per minute per domain
• Adjust crawl speeds when encountering server errors (4xx, 5xx)
• You can find more information on our content classification here

Value for Website Owners

Website owners can benefit from Scope3's content classification pipeline in the following ways:

• Verify domain ownership through a free Scope3 publisher account at scope3.com, gaining complete visibility into brand safety and vertical model classifications.
• Implement brand safety targeting with our integration tools. Publishers can use our integration product to remove the need for crawling.

Monitoring & Support

For questions about crawl frequency or suspicious traffic, contact us at [email protected]. We're committed to addressing your concerns and finding suitable solutions.