tzuintel

tzuintel is the crawler behind Tzu, which maps the companies in a market from their public pages. It identifies itself as tzuintel and links here.

What it reads

Public pages a company publishes about itself: its homepage, pricing, changelog, careers and similar pages. It never logs in, never submits forms, and never reads anything behind a paywall or a login. It keeps a dated copy of each page it reads, so every value Tzu shows can be traced to the page it came from.

How it behaves

• It reads and obeys robots.txt, under the tzuintel user agent or *.
• It makes at most one request every two seconds to a host, or slower if your Crawl-delay asks for it.
• If your robots.txt can't be read because of a server error, it doesn't crawl.

Opting out

Add this to your robots.txt:

User-agent: tzuintel
Disallow: /

tzuintel re-reads robots.txt at least once a day, so a change takes effect within a day.