# Robots.txt # Allow all robots to spider the domain User-agent: * Disallow: # Disallow directories User-agent: * Disallow: /_assets/server/ Disallow: /oldsite/ sitemap: http://www.example.com/sitemap.xml