# Robots.txt # Allow all robots to spider the domain User-agent: * Disallow: # Disallow directories User-agent: * Disallow: /_assets/server/ Disallow: /oldsite/ Disallow: /admin/ sitemap: http://www.doverport.co.uk/sitemap.xml