User-agent: Scrapy Allow: / # Termly scanner User-agent: Scrapy/2.5.1 (+https://scrapy.org) Allow: /