#accept all spiders User-agent: * # disallow index core root folders/files Disallow: /bin Disallow: /Channels Disallow: /Classes Disallow: /Common Disallow: /css Disallow: /Holding Disallow: /info Disallow: /Interfaces Disallow: /javascript Disallow: /masterpages Disallow: /obj Disallow: /Routing Disallow: /Secure Disallow: /UserControls Disallow: /EmailAFriendPopUp.aspx Disallow: /Error.aspx Disallow: /Global.asax Disallow: /Redirect.aspx Disallow: /ViewBasket.aspx Disallow: /web.config