Managing robots.txt using PHP: Generating Dynamic Syntax - Upload the complete myrobots.txt.php to the website's root directory

The complete/working myrobots.txt.php script is shown below:

<?php

header("Content-Type: text/plain");

$currentsyntax='User-agent: *

Disallow: */trackback

Disallow: */feed

Disallow: /searchresultpages

Disallow: /wp-

Disallow: /*?

Disallow: /xmlrpc.php

Disallow: /blockedbyrobots.php

Allow: /wp-content/uploads/scripts/PHP-Server-Array-Variables.php

Disallow: /postpdfcreator.php

Disallow: /search/

Disallow: /search

Disallow: /*.js$

Disallow: /antibot.php

Disallow: /ajaxwebform/captcha.php

Disallow: /*.jpg$

Disallow: /ajaxwebform/ajaxvalidate.php

Disallow: /hiddentextexample.php

Disallow: /searchresultpages/

Allow: /index.php?page_id=123&pg=2

Allow: /site-map/?pg=2';

echo $currentsyntax;

/////////////////////////////////////////////////////////////////////////////

//DYNAMICALLY GENERATE ROBOTS.TXT SYNTAX FOR YEAR DIRECTORY//////////////////

////////START////////////////////////////////////////////////////////////////

//connect to MySQL database

$username = "xxxx";

$password = "xxxxx";

$hostname = "xxxx";

$database = "xxxxx";

$dbhandle = mysql_connect($hostname, $username, $password)

or die("Unable to connect to MySQL");

$selected = mysql_select_db($database,$dbhandle)

or die("Could not select $database");

//Retrieve latest post dates in wp_post column

$latestpostdate = mysql_query("SELECT max(post_date) from `wp_posts`") or die(mysql_error());

$rowlatestpostdate = mysql_fetch_array($latestpostdate) or die("Invalid query" . mysql_error());

$latestpostdatedata = $rowlatestpostdate['max(post_date)'];

//Retrive the 1st post date in wp_post column

$firstpostdate = mysql_query("SELECT min(post_date) from `wp_posts`") or die(mysql_error());

$rowfirstpostdate = mysql_fetch_array($firstpostdate) or die("Invalid query" . mysql_error());

$firstpostdatedata = $rowfirstpostdate['min(post_date)'];

//Extract the post year of both 1st and latest post

$firstdashlatest = strpos($latestpostdatedata,"-");

$countlatestyear=strlen($latestpostdatedata);

$substrcountlatest= ($countlatestyear-$firstdashlatest)*(-1);

$latestpostyear = substr($latestpostdatedata,0,$substrcountlatest);

$firstdashfirst = strpos($firstpostdatedata,"-");

$countfirstyear=strlen($firstpostdatedata);

$substrcountfirst= ($countfirstyear-$firstdashfirst)*(-1);

$firstpostyear = substr($firstpostdatedata,0,$substrcountfirst);

$i=$firstpostyear;

echo strip_tags(nl2br("rn# Start of dynamically generated robots.txt syntax"));

while ($i<=$latestpostyear) {

echo strip_tags(nl2br("rnDisallow: /".$i++."/"));

}

echo strip_tags(nl2br("rn# End of dynamically generated robots.txt syntax"));

/////////////////////////////////////////////////////////////////////////////

////////END OF DYNAMIC ROBOTS.TXT SYNTAX GENERATION//////////////////////////

/////////////////////////////////////////////////////////////////////////////

echo strip_tags(nl2br("rn"));

echo strip_tags(nl2br("rnSitemap: http://www.php-developer.org/sitemap.xml"));

?>

Next: Revise .htaccess to rewrite myrobots.txt.php to robots.txt >>
 

More PEAR Articles Articles
More By Codex-M