To gather keywords for our search application, we will provide a simple form that will accept the URLs of the websites to be searched. This will be effective for small sites where all the URLs are known. Large sites with dynamic URLs would be better served by a spider script. Writing an effective spider is well beyond the scope of this document and will be left for an exercise to the reader. For reference, you might want to look at the PHPDig project as they have nice spider built into their application.
The harvest.php script itself will not have much PHP code in it. The reason for this is that the code to perform the harvesting functions will be in a class that is stored in a file named harvestclass.php. Let's take a high level overview of the Harvest_Keywords class and then we will examine each function in detail.
Declare class variables - The first order of business in the class is to declare the variables that will be used to store information that will be shared throughout the class.
Constructor - The constructor of this class will first establish a database connection utilizing the database class we created earlier. Then it will store, as an array, the URL(s) that were entered.
_prune - This function is a private function of the class. A private function is one that is not called from outside the class. PHP does not provide a way to truly make a function private, but we will precede any function we deem to be private with an underscore. This just serves to document them as private for our later reference. We will use this function in conjunction with the array_walk function of PHP to remove unwanted words from the keywords we gather.
_checkURL - Also a private function, this will run a perform a very basic validation of a URL.
_getData - We will use this private function to take care of the details of opening a connection to a URL and obtaining the source.
_harvest - This function is where the real work is done. This private function will call the other functions of this class to gather all the keywords from a single URL and store them in a database.
process - This function is will iterate through each URL provided to it and call the _harvest function to gather keywords.
