You are here

Home » API reference » Zircon Profile 8

robots.txt in Zircon Profile 8

Same filename and directory in other branches
  1. 8.0 robots.txt
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: *
# JS/CSS
Allow: /core/*.css$
Allow: /core/*.js$
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.txt
Disallow: /web.config
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips/
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password/
Disallow: /index.php/user/register/
Disallow: /index.php/user/login/
Disallow: /index.php/user/logout/

File

robots.txt
View source 
  1. #

  2. # robots.txt

  3. #

  4. # This file is to prevent the crawling and indexing of certain parts

  5. # of your site by web crawlers and spiders run by sites like Yahoo!

  6. # and Google. By telling these "robots" where not to go on your site,

  7. # you save bandwidth and server resources.

  8. #

  9. # This file will be ignored unless it is at the root of your host:

  10. # Used:    http://example.com/robots.txt

  11. # Ignored: http://example.com/site/robots.txt

  12. #

  13. # For more information about the robots.txt standard, see:

  14. # http://www.robotstxt.org/robotstxt.html

  15. 

  16. User-agent: *

  17. # JS/CSS

  18. Allow: /core/*.css$

  19. Allow: /core/*.js$

  20. # Directories

  21. Disallow: /core/

  22. Disallow: /profiles/

  23. # Files

  24. Disallow: /README.txt

  25. Disallow: /web.config

  26. # Paths (clean URLs)

  27. Disallow: /admin/

  28. Disallow: /comment/reply/

  29. Disallow: /filter/tips/

  30. Disallow: /node/add/

  31. Disallow: /search/

  32. Disallow: /user/register/

  33. Disallow: /user/password/

  34. Disallow: /user/login/

  35. Disallow: /user/logout/

  36. # Paths (no clean URLs)

  37. Disallow: /index.php/admin/

  38. Disallow: /index.php/comment/reply/

  39. Disallow: /index.php/filter/tips/

  40. Disallow: /index.php/node/add/

  41. Disallow: /index.php/search/

  42. Disallow: /index.php/user/password/

  43. Disallow: /index.php/user/register/

  44. Disallow: /index.php/user/login/

  45. Disallow: /index.php/user/logout/