Bot policy
WorthabidBot
If you run a tender portal or a business website and have seen this user agent in your logs, this page explains what it was doing and how to stop it.
User agent
WorthabidBot/1.0 (+https://worthabid.com.au/bot)
What it fetches
Public tender portals. Listing and notice pages for open tenders, awarded contracts and forward procurement plans. We store the notice metadata (title, buyer, category, value, dates) and a link back to the original. We never download tender attachments and never reproduce tender documents.
A customer’s own website. When a business signs up and gives us its website address, the bot reads that site’s public pages once to draft the business’s capability profile. It fetches only the site the customer named, a few dozen pages at most, and stops there.
It does not fetch anything behind a login, does not submit forms, and does not follow links off the site it was asked to read.
How it behaves
- Obeys robots.txt, including crawl-delay, on every host. Disallow the user agent and it stops.
- At most one request every three seconds per host.
- Sends If-None-Match and If-Modified-Since so unchanged pages are not transferred again.
- Backs off with jitter on 429 and 5xx responses.
- Keeps fetched HTML for 30 days for reprocessing, then deletes it.
- Requests come from Australian addresses only.
How to opt out
Add this to your robots.txt and the bot will stop on its next visit.
User-agent: WorthabidBot
Disallow: /Or email bot@worthabid.com.au with the host name and we will add it to the exclusion list by hand within two business days and confirm by reply. The same address is the place for questions about a specific request you have seen in your logs.
The full list of portals we read, and which ones we have excluded and why, is on the sources page.