DataScoutBot
DataScoutBot is the web crawler behind Data Scout, operated by GAMMA IT SOLUTIONS SRL. When one of our customers looks up a company, it reads that company's public website and turns it into a report: company details, technology stack, an SEO audit and published contact addresses.
What it does
- User-Agent
- DataScoutBot/1.0 (+https://getdatascout.com/bot)
- What it fetches
- Your robots.txt, your landing page, and up to four pages it ranks as likely contact or imprint pages — about six requests for a typical analysis. A full site scan, which a customer starts deliberately, reads at most 20 pages on the same host and stops at a byte budget.
- How it fetches
- Plain HTTP GET, public pages only. It does not run JavaScript, submit forms, sign in, or follow anything behind a login. Each request times out after 10 seconds.
- How often
- Rarely. Results are cached per domain for up to 30 days and shared across all our customers, so a hundred people looking your company up is still one visit.
- If you throttle it
- A 429 is taken at face value: it waits for your Retry-After, or a few seconds when you send none, and gives up rather than hammering.
- What we keep
- Business information published on the pages themselves — company details, technologies, SEO signals, and business contact addresses. Nothing behind a login, and no personal data beyond what you publish as a way to be contacted.
It honours robots.txt
Before anything else, it reads your /robots.txt. If it is disallowed there,
the analysis is refused outright and your site is not fetched — and every additional
page it considers is checked against your rules again, on that page's own host.
To keep it out entirely, add this to your robots.txt:
User-agent: DataScoutBot
Disallow: /The change takes effect on our next visit. Blocking the User-Agent at your WAF or CDN works too — we would simply rather you did not have to.
Talk to a person
If the crawler misbehaved, or you want your company's data removed from our index rather than just the crawling stopped, write to abuse@getdatascout.com — a real mailbox, read by the people who run the crawler. For anything else, support@getdatascout.com.