You are here

Home » API reference » RobotsTxt 7

README.txt in RobotsTxt 7

Same filename and directory in other branches
  1. 5 README.txt
  2. 6 README.txt
CONTENTS OF THIS FILE
---------------------

 * Introduction
 * Installation
 * Frequently Asked Questions (FAQ)
 * Known Issues
 * How Can You Contribute?


INTRODUCTION
------------

Maintainer: hass <https://drupal.org/user/85918>
Project Page: https://drupal.org/project/robotstxt

Use this module when you are running multiple Drupal sites from a single code
base (multisite) and you need a different robots.txt file for each one. This
module generates the robots.txt file dynamically and gives you the chance to
edit it, on a per-site basis.

For developers, you can automatically add paths to the robots.txt file by
implementing hook_robotstxt(). See robotstxt.api.php for more documentation.


INSTALLATION
------------

See https://drupal.org/getting-started/install-contrib for instructions on
how to install or update Drupal modules.

Once you have the RobotsTxt modules installed, make sure to delete or rename
the robots.txt file in the root of your Drupal installation. Otherwise, the
module cannot intercept requests for the /robots.txt path.


FREQUENTLY ASKED QUESTIONS
--------------------------

Q: Can this module work if I have clean URLs disabled?
A: Yes it can! In the .htaccess file of your Drupal's root directory, add the
   following two lines to the mod_rewrite section, immediately after the line
   that says "RewriteEngine on":

   # Add redirection for the robots.txt path for use with the RobotsTxt module.
   RewriteRule ^(robots.txt)$ index.php?q=$1

Q: Does this module work together with Drupal Core "Fast 404 pages" feature?
A: Yes, but you need to add robots.txt to the 'exclude_paths' of your 
   settings.php.
   
   Drupal (default):
   $conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\//';

   Drupal with RobotsTxt module:
   $conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\/|(?:robots.txt)/';

Q: How can I install the module with custom default robots.txt?
A: The module allows adding a default.robots.txt to the defaults folder.

   1. Remove the robots.txt from site root.
   2. Save your custom robots.txt to "/sites/default/default.robots.txt"
   3. Run the module installation.


KNOWN ISSUES
------------

There are no known issues at this time.

To report new bug reports, feature requests, and support requests, visit
https://drupal.org/project/issues/robotstxt.


HOW CAN YOU CONTRIBUTE?
---------------------

- Report any bugs, feature requests, etc. in the issue tracker.
  https://drupal.org/project/issues/robotstxt

File

README.txt
View source 
  1. 

  2. CONTENTS OF THIS FILE

  3. ---------------------

  4. 

  5.  * Introduction

  6.  * Installation

  7.  * Frequently Asked Questions (FAQ)

  8.  * Known Issues

  9.  * How Can You Contribute?

  10. 

  11. 

  12. INTRODUCTION

  13. ------------

  14. 

  15. Maintainer: hass 

  16. Project Page: https://drupal.org/project/robotstxt

  17. 

  18. Use this module when you are running multiple Drupal sites from a single code

  19. base (multisite) and you need a different robots.txt file for each one. This

  20. module generates the robots.txt file dynamically and gives you the chance to

  21. edit it, on a per-site basis.

  22. 

  23. For developers, you can automatically add paths to the robots.txt file by

  24. implementing hook_robotstxt(). See robotstxt.api.php for more documentation.

  25. 

  26. 

  27. INSTALLATION

  28. ------------

  29. 

  30. See https://drupal.org/getting-started/install-contrib for instructions on

  31. how to install or update Drupal modules.

  32. 

  33. Once you have the RobotsTxt modules installed, make sure to delete or rename

  34. the robots.txt file in the root of your Drupal installation. Otherwise, the

  35. module cannot intercept requests for the /robots.txt path.

  36. 

  37. 

  38. FREQUENTLY ASKED QUESTIONS

  39. --------------------------

  40. 

  41. Q: Can this module work if I have clean URLs disabled?

  42. A: Yes it can! In the .htaccess file of your Drupal's root directory, add the

  43.    following two lines to the mod_rewrite section, immediately after the line

  44.    that says "RewriteEngine on":

  45. 

  46.    # Add redirection for the robots.txt path for use with the RobotsTxt module.

  47.    RewriteRule ^(robots.txt)$ index.php?q=$1

  48. 

  49. Q: Does this module work together with Drupal Core "Fast 404 pages" feature?

  50. A: Yes, but you need to add robots.txt to the 'exclude_paths' of your 

  51.    settings.php.

  52.    

  53.    Drupal (default):

  54.    $conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\//';

  55. 

  56.    Drupal with RobotsTxt module:

  57.    $conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\/|(?:robots.txt)/';

  58. 

  59. Q: How can I install the module with custom default robots.txt?

  60. A: The module allows adding a default.robots.txt to the defaults folder.

  61. 

  62.    1. Remove the robots.txt from site root.

  63.    2. Save your custom robots.txt to "/sites/default/default.robots.txt"

  64.    3. Run the module installation.

  65. 

  66. 

  67. KNOWN ISSUES

  68. ------------

  69. 

  70. There are no known issues at this time.

  71. 

  72. To report new bug reports, feature requests, and support requests, visit

  73. https://drupal.org/project/issues/robotstxt.

  74. 

  75. 

  76. HOW CAN YOU CONTRIBUTE?

  77. ---------------------

  78. 

  79. - Report any bugs, feature requests, etc. in the issue tracker.

  80.   https://drupal.org/project/issues/robotstxt