Identity and purpose
Product token: WebsiteIQBot
User-Agent: WebsiteIQBot/1.2 (+https://websiteiq.xyz/crawler/)
Operator: Osborn Ventures Inc.
Contact: WebsiteIQ contact form
WebsiteIQBot performs a user-requested diagnostic audit of public website delivery, metadata, structured data, content structure, security headers, discovery surfaces, and related evidence.
robots.txt behavior
- Validate the submitted public HTTP(S) origin.
- Fetch
/robots.txtbefore any content resource. - Select exact case-insensitive
WebsiteIQBotgroups; use*only if no exact group exists; merge matching groups. - Apply longest-match rules with Allow winning equal-specificity ties, including
*and terminal$. - Honor the matching
Crawl-delayconservatively even though it is not part of RFC 9309.
An ordinary robots 4xx other than 401, 403, or 429 is treated as unavailable. Network failure, 401, 403, 429, and 5xx stop the crawl fail-closed. If the submitted page is disallowed, no audit crawl occurs.
Request behavior
| Boundary | Current public-beta behavior |
|---|---|
| Pages | The new free beta sample audits one permitted page per request through the website, API, or MCP. A valid email and explicit audit-processing consent are required. Future paid API/MCP limits will be published only when x402 settlement goes live. An inline audit stops before content when the matching robots crawl delay exceeds 10 seconds. |
| Pacing | Sequential requests, at least 750 ms apart, or the matching larger crawl delay. A redirected www/apex host gets its own robots check before content. |
| Hosts | Submitted host and its direct www/apex counterpart only. Redirects are revalidated. |
| Ports | 80 and 443 only. |
| Networks | Private, loopback, link-local, reserved, multicast, unresolved, and mixed public/private DNS targets are rejected. |
| Bodies | Bounded at 2,000,000 bytes per target response; text retained only to the configured analysis limit. |
| Credentials | No cookies, target authorization, embedded URL credentials, or target login. |
| Payment | HTTP 402 stops the request. WebsiteIQBot never pays for a crawl or retries with a payment payload. |
Secondary requests
Sitemaps, llms.txt, and discovered same-host pages are requested only when permitted. The Python report engine inventories third-party asset references from markup but does not contact those third parties for page-weight probes. The inline Worker does not probe third-party assets.
How to control WebsiteIQBot
User-agent: WebsiteIQBot Disallow: /
To permit a section, use an Allow rule more specific than the applicable Disallow. To set a delay, place Crawl-delay: N in the selected group. WebsiteIQ does not treat robots as authorization or confidentiality; protect private resources with real access control.
Questions or complaints
Use the contact form with the target hostname, UTC timestamp, affected path, and relevant server-log line. Do not include credentials, report access tokens, or unrelated submission content.