User-agent: Scrapy Allow: / # Robots.txt file updated 27/03/20 User-agent: * Disallow: /*/feed/ sitemap: https://www.newtons.co.uk/sitemap_index.xml