xbluBot

xbluBot is the crawler of Xblu's market intelligence engine. It reads the public feeds, pages and data of statistics offices, central banks, regulators, exchanges and publishers, to follow the events that move the markets Xblu lists.

How it names itself

Every request it sends carries this user agent:

xbluBot/1.0 (+https://xblu.io/bot; tech@xblu.io)

How it reads a site

• It reads robots.txt as xbluBot, once a day per site, and never asks for a path robots.txt disallows. The one exception is an API its operator documents for programs, such as a statistics office's data API, which it calls within the limits that operator publishes.
• It asks one site for one thing at a time, at least a second apart or as far apart as the crawl delay robots.txt sets, and waits out a Retry-After.
• It sends back the ETag and the Last-Modified it last received, so a page that has not changed costs a 304.
• A site whose robots.txt signals ai-input=no in its Content-Signal, or a page reserved from text and data mining, is kept away from any AI processing.

How to stop it

Add these two lines to your robots.txt. xbluBot reads it again within a day.

User-agent: xbluBot
Disallow: /

Or write to tech@xblu.io, and we stop reading your site.