Managing robots.txt using PHP: Generating Dynamic Syntax - Revise .htaccess to rewrite myrobots.txt.php to robots.txt

This is the standard .htaccess file for WordPress,

# BEGIN WordPress

<IfModule mod_rewrite.c>

RewriteEngine On

RewriteBase /

RewriteCond %{REQUEST_FILENAME} !-f

RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule . /index.php [L]

</IfModule>

You need to add this rewrite rule to the existing .htaccess lines:

RewriteEngine On

RewriteRule ^robots.txt$ /myrobots.txt.php

So it will look like this:

This rewrite rule will basically assign the contents of myrobots.txt.php to robots.txt.

You will need to upload the .htaccess to the root directory of your server, in the same path as with the myrobots.txt.php file.

Implementation Technique

It is important to note several important points when implementing this project.

Do not remove or delete the old robots.txt file; it should still reside in the server root directory. It is okay to erase all the contents of the old robots.txt file, but never delete it.

Therefore the following files must be in the server root directory for this method to work:

a. .htaccess - this contains the rewrite rule from myrobots.txt.php to robots.txt

b. "old" robots.txt - this is the old robots.txt file which you were using previously (before you started using PHP to generate syntax).

c. myrobots.txt.php - this is the actual script that generates the robots.txt syntax.

A full sample implementation of this project can be seen at http://www.php-developer.org/robots.txt . The robots.txt is dynamically generated using PHP, as indicated by these comments in the robots.txt syntax:

# Start of dynamically generated robots.txt syntax

Disallow: /2009/

Disallow: /2010/

# End of dynamically generated robots.txt syntax

Finally, for safety reasons, it is recommended that you use another name for the PHP file generating your syntax, and not the one used in this tutorial, which is myrobots.txt.php