Malt & PressA visiting indexMalt & Press

MaltAndPressBot

MaltAndPressBot is the crawler behind Malt & Press, a free multilingual directory of producer visits and tastings across Europe. It only visits websites of producers that are already listed on Malt & Press, to keep listings accurate: opening status, visits and tastings offered, prices, and tour languages.

How it behaves

  • Honors robots.txt, including Crawl-delay, for both MaltAndPressBot and * groups.
  • Rate-limited: roughly one request per second per domain, with a 15 second timeout per request.
  • Strict size caps: at most 2 MB fetched per page, and at most 400 KB of text retained for processing.
  • Fetched pages are internal working copies used to extract facts; they are never republished.
  • Identifies itself in every request: MaltAndPressBot/0.1 (+https://maltandpress.com/bot).

Blocking or questions

To opt out entirely, add this to your robots.txt:

User-agent: MaltAndPressBot
Disallow: /

Questions, corrections, or removal requests: see the contact page. Your producer’s listing page is the fastest route, and every request is reviewed by a person.