FirmtrailBot

How to recognise it

We renamed CompanyIntelBot to FirmtrailBot; some requests may still identify as CompanyIntelBot/1.0. The crawler carries this User-Agent, with a link to this page:

FirmtrailBot/1.0 (+https://firmtrail.io/crawler)

What it fetches

• Public job board APIs that applicant tracking systems publish for this purpose (Greenhouse, Lever, Ashby, Recruitee, Personio, Workable), about once a day per board.
• A few public pages of a company’s own website: the home page, pages such as About, Products, Pricing and Careers, robots.txt and sitemap.xml. Never pages behind a login. Each visit has a fixed cap (requests, not counting robots.txt):
  • a company profile: at most 8;
  • looking for a company’s job board: at most 15;
  • a careers page with no job board API: the page and at most 20 job pages, once a day.
• Public DNS records of the company’s domain (MX and TXT), to tell which services a company uses.

It does not fetch LinkedIn, Indeed or Glassdoor. It does not run JavaScript, submit forms or follow links to other sites.

How it behaves

• It obeys robots.txt for FirmtrailBot, for CompanyIntelBot (so an existing opt-out still applies) and for *.
• It sends at most one request per second to any one host. It backs off on 429 and 503 and honours Retry-After.

How to stop it

Add both entries to your site’s robots.txt. It takes effect within a day:

User-agent: FirmtrailBot
Disallow: /

User-agent: CompanyIntelBot
Disallow: /

To have your company or job board left out entirely, email support@firmtrail.io from an address at your company’s domain. We delete everything stored for it, including its postings and their history, and we do not collect it again. Copies in our backups are gone within about 14 weeks; operational logs that name the domain age out with our log rotation, and within 14 days at our monitoring provider.

Contact

For the crawler, removal requests, privacy questions and API support: support@firmtrail.io.