README.txt in Recommender API 7.3

=============================
Recommender API Documentation
=============================

:Authors:
	Daniel Xiaodan Zhou (http://michiza.com/)
:Version:
	For Recommender API 7.x-3.x
:Note:
	The HTML version of this document is generated by docutils.


.. contents::


What is Recommender API?
========================

Recommender API provides easy-to-understand, easy-to-use, fully-documented APIs to write Drupal content recommendation modules (e.g., Fivestar Recommender, Ubercart Product Recommender, etc.). It also provides a unified approach to configure & execute the recommender algorithms, and to display results to end users.


What's new in 7.x-3.x?
----------------------

The 3.x release completely re-writes all API code, and is not compatible with earlier releases. Major changes and features are:

	* All complex computation is now done under Apache Mahout (http://mahout.apache.org/), which is much faster than the original PHP implementation. See http://drupal.org/node/1180000 for more details.
	* All end user display is now through customizable Views rather than hard coded blocks. See http://drupal.org/node/673786 for more details.
	* You can isolate the resource intensive recommender computation from the Drupal production site.
	* It now supports more algorithms (e.g. SVD) provided by Apache Mahout.




Installation, Configuration & Execution 
=======================================

How does Recommender API work?
------------------------------

Conceptually, you need 2 computers to run Recommender API: the Drupal server for your Drupal site, and the recommender server to compute recommendations. Of course, you can run the recommender server on the same Drupal server if you like, but note that the recommender computation could easily consume all resources. On the Drupal server, you simply issue a command to run recommenders. On the recommender server, you would run the real recommender program (written in Java), which takes the command, computes results, and saves the results back to the Drupal server database for display.


Out-of-the-box Installation
-----------------------------

This simplified installation process assumes you have Recommender API installed under "sites/all/modules/recommender" on your Drupal server. If you want to offload recommendations computations to a separate recommender server, please follow the generic installation process in the next section.

**Step 1**. Install the Recommender API module to "sites/all/modules/recommender".

**Step 2**. Download Apache Mahout v.0.5+ from http://mahout.apache.org/, and extract it to "sites/all/modules/recommender/mahout".

**Step 3**. Install the Async Command module (http://drupal.org/project/async_command) to "sites/all/modules/async_command".

**Step 4**. Install any helper modules to your Drupal server, such as Browsing History Recommender, Fivestar Recommender, Ubercart Product Recommender, etc.

**Step 5**. Periodically execute "run.sh" (Linux/Mac/Unix) or "run.bat" (Windows) under "sites/all/modules/recommender" to compute recommendations. You might need to edit "run.sh" or "run.bat" and change the DRUPAL_HOME parameter in case of errors.



Installation (Generic Process)
-------------------------------

(You can skip this section if you use the out-of-the-box installation process.)

**Step 1**. Download Apache Mahout v.0.5+ from http://mahout.apache.org/, and extract it to any directory on the recommender server.

**Step 2**. Install the Async Command module (http://drupal.org/project/async_command) to your Drupal server under sites/all/modules/async_command. Copy the 'lib' sub-directory to any folder on the recommender server.

**Step 3**. Install the Recommender API module (http://drupal.org/project/recommender) to your Drupal server under sites/all/modules/recommender. Copy the 'recommender.jar' file to any folder on the recommender server.

**Step 4**. Install any helper modules to your Drupal server, such as Browsing History Recommender, Fivestar Recommender, Ubercart Product Recommender, etc.

**Step 5**. On your recommender server, create the 'config.properties' file similar to 'config.properties.example'. This file specifies how the recommender program ("recommender.jar") access your Drupal database. The  access doesn't need full DB privileges. The minimum privileges are::

	SELECT/UPDATE on {async_command}, {recommender_app}
	SELECT on {node} (nid, created), {users} (uid, created), or other tables (see documentation of relevant helper modules)
	SELECT/INSERT/UPDATE/DELETE on {recommender_similarity}, {recommender_prediction}, {recommender_preference_staging}

**Step 6**. Copy 'run.sh' (Linux/Mac/Unix) or 'run.bat' (Windows) to any directory on the recommender server, and manually set the parameters to point to the correct directories. 

**Step 7**. Periodically execute 'run.sh' or 'run.bat' on your recommender server to compute recommendations.



Configuration & Execution
------------------------------

On your Drupal server, you can go to admin->configure->recommender to run various recommenders. Note that this doesn't do any computation, but merely issues the commands in the {async_command} queue to be executed on your recommender server. Remember to set "administer recommender" permission too.

On your recommender server, you need to periodically execute 'run.sh' or 'run.bat' to actually compute the recommendations. If you use Linux, you might want to run 'run.sh' as a cron job on your recommender server. This is different from the cron settings on the Drupal server, which just issues the commands. The cron settings could be like this (run every 30 minutes)::

	# in crontab -e, add the following line.
	*/30 * * * * flock -n /tmp/recommender.lock recommender/run.sh >> /tmp/recommender.log 2>&1



Similarity vs. prediction
--------------------------

Recommender API offers two types of recommendations, similarity-based and prediction-based, although different algorithms might implement both or either (e.g. SlopeOne algorithm only has prediction-based recommendations).

One type of recommendations is based on the similarity among nodes (or users, or other types of entities). For example, if you are viewing a node, it will recommend other similar nodes. The recommended nodes are the same for this particular node regardless of which user is viewing it. The similarity scores are computed based on the fact that, for example, if two nodes are usually viewed together, or two products usually purchased together, then the two nodes/products are similar. The helper modules actually define what information to use to compute the similarity scores. The similarity scores range from -1 (completed dissimilar) to +1 (completely similar), and are directional: A is similar to B doesn't mean B is similar to A.

The other type of recommendations is based on the "prediction scores", which predict how much a user would like a node. The recommendations are personalized: different users would see different recommendations. But for each user, she would see the same recommendations regardless which page she is viewing. The prediction scores are computed based on the user's personal history. For example, if a user purchased products A and B, she might be interested in purchasing C which is similar to A and B. Exactly what "personal history" to use is defined by the helper modules. (A side note: If you treat users as nodes and nodes as users, you can then predict how much a node would "like" a user. This is useful when you want to promote a node to the most interested users.)

You need to understand the distinction between similarity and prediction in order to work with Views.



Views support
----------------------

Recommender API supports Views 3, which is the preferred way to display recommendations. Most helper modules would create default Views, and you can just customize those. 

However, if you do want to create your own recommender views, here's how:

**Step 1**: Choose the views base table, either Recommender Similarity or Recommender Prediction, depending on which type of recommendations you would show.

**Step 2**: In "filter criteria", you need to select which recommender application to provide the recommendations. Usually you just need the "Application ID" filter (if you are a helper module developer, please use "Application Name" filter). Use other filters if you want.

**Step 3**: In "relationships", add a new "Entity ID (Target)" relationship. In the next page, select the entity type of the recommended items. For example if your recommendations are nodes, then use "Content". Also check the "Require this relationship".

**Step 4**: In "contextual filters" (a.k.a. "Arguments"), add "Entity ID (Source)". This is where the recommendations are made for. If your recommendations are made for the current user, then here is the UserID of the current user. Usually you want provide a default value of either the current node or the current user.

**Step 5**: Add "fields", "sort criteria", or make other Views settings as you see fit. When you sort by similarity scores or prediction scores, choose "descending".




Recommender Algorithms Explained
================================

User-user vs. item-item
------------------------

The two most popular recommender algorithms are user-user and item-item. The user-user algorithm first computes similarities among users based on the users history records (such as purchasing history, nodes browsing history, etc), and then predicts how much a user likes an item based on how much the user's similar users like the item. The item-item algorithm first computes similarities among items based on some information (e.g., the items are always purchased together, the items are always rated the same scores, etc.), and then predicts how much a user likes an item based on how much the user likes the item's similar items.

Academic research shows that the item-item algorithm usually works better than user-user. Amazon.com uses the item-item algorithm in its recommender system. 


SlopeOne
-----------

The advantage of SlopeOne is performance. But it doesn't compute similarity scores, and I don't know many real systems use this algorithm. (Note: This algorithm will be added later.)


SVD
-----------

This algorithm worked really well in the Netflix Prize (http://www.netflixprize.com/). It is especially useful when you have sparse datasets. (Note: This algorithm will be added later.)



For Developers
========================

In the simplest case, there are 2 steps to use Recommender API. First, use "recommender_app_register()" to register your recommender application with the system (see parameters descriptions in the sub-section). Second, use "recommender_create_command()" to issue an execution command to compute recommendations.

You can use "recommender_app_update()" to update your recommender application's parameters after the app is registered. In "hook_disable()" or "hook_uninstall()" of your module, you would use "recommender_app_unregister()" to clean up the data for your recommender application.

You would use "$app_name", a machine readable string, to identify your recommender application for the recommender_app_*() functions. You can use "recommender_app_load()" to retrieve the numeric ID used internally for the recommender applications.

To display the recommendations, the suggested approach is by using Views. You can create default Views for your module using the "export" UI command in the Views module, and then copy the exported code to "yourmodule_views_default_views()". You can also use "recommender_default_view_template()" to easily generate default views for your module. In addition, you can use "recommender_top_similarity()" and "recommender_top_prediction()" to retrieve a list of recommendations programatically.

The best way to learn how to program with Recommender API is through examples. You can read the code of "rec_example" module that comes with this module, or read the code of other recommender modules (eg, history_rec, fivestar_rec, etc). To see how to use the API functions, read the comments in recommender.module. If you need further support, please create issues in the module's issue queue.



Parameters for recommender_app_register($apps)
-------------------------------------------------

$apps is an array of recommender applications, where the key is a machine readable string to identify a recommender application::

	$apps = array(
		$app_name1 => $app_definition1,
		$app_name2 => $app_definition2,
		...
	);

$app_definition is an array defined as follows::
	
	$app_definition = array(
		$title => 'My Recommender App', // human readable name for the app
		$params => array(...),
	); 

$params is an array for your recommender application's parameters. The fields are as follows:

- 'algorithm' (required): You can choose from the following recommender algorithms:
	- item2item: The most commonly used algorithm.
	- item2item_increment: Incremental update to a base item2item application.
	- user2user: The user-user algorithm.
	- svd: The SVD algorithm.

- 'table' (required if no 'sql'): This specifies the user-item preference table. For example, in history_rec, this is the {history} table that logs which users browsed which nodes. The table needs to be enclosed with {}. Required if 'sql' is not set.

- 'sql' (required if no 'table'): You can use an alternative SQL statement to generate the preference table dynamically if you don't use the 'table' field. Required if 'sql' is not set.

- 'fields' (required): This is an array that specifies the required column names for 'table' or 'sql': array(user_id, item_id, [preference], [timestamp]), where [preference] and [timestamp] can be omitted.

- 'similarity' (optional): You can select which similarity algorithm to use: 'auto' (default), 'cityblock', 'euclidean', 'loglikelihood', 'pearson', 'spearman', 'tanimoto', and 'cosine'. To see the explanation of each algorithm, go to http://goo.gl/E85TR

- 'performance' (optional): You can choose 'database' to run the recommender application in the database (saves RAM), or 'memory' to pre-load all data into memory (improves speed), or the default 'auto' to let the program decide.

- 'preference' (optional): Specifies whether the preference is a 'boolean' (true or false) or a 'score' (1-5 score). In history_rec where preference is whether a user browsed a node or not, this should be 'boolean'. In fivestar_rec where preference is a 1-5 score, this should be 'score'.

- 'max_keep' (optional): Specifies the maximum number of similarity/prediction records to keep in the database for each user or item. Default is 100.

- 'base_app_name' (required if 'algorithm' is 'item2item_increment'): Specifies the $app_name of the base item2item application upon which the incremental update is for.




FAQ
===========

Why not using REST to access Apache Mahout?
--------------------------------------------

Apache Mahout provides REST access. However, this module choose not to use it for the following reasons:

* Each recommender application (Fivestar Recommender, Browsing History Recommender, etc.) would require an independent Mahout REST instance, which involves lots of administration overhead.
* Even though we can use the REST interface to query recommendations, Mahout still requires direct database access through its JDBCDataModel.
* The recommender algorithms usually requires access to the entire database tables all at once. It's much more efficient using direct database access than using REST.


What happens to the mouse/cheese metaphor used in the 6.x-2.x release?
-----------------------------------------------------------------------

The mouse/cheese metaphor was used for two reasons. First, it's more lively than the user/item terminology. Second, from a programming perspective, users and items are usually inter-changeable. So a "mouse" can act as a user at one time but as an item at another time, same for "cheese". But Mahout adopts the user/item terminology, and it handles the user/item inter-changeability by class hierarchy. To avoid confusion in 3.x, I'm not using the mouse/cheese metaphor anymore.


Is there a cloud service alternative?
--------------------------------------

We will launch a cloud service shortly. 



Where to find more documentation and support?
---------------------------------------------

The HTML version of this documentation is posted at http://drupal.org/node/1207634, but will not get updated as frequently as this one. You can use rst2html to generate HTML too.

For bugs report, new features requests and all other requests, please submit issues at http://drupal.org/project/issues/recommender.

If you need customization or consulting services, please contact the author at danithaca@gmail.com.