You are here

README.txt in Feeds 7.2

Same filename and directory in other branches
  1. 8.3 README.txt
  2. 8.2 README.txt
  3. 6 README.txt
  4. 7 README.txt

"It feeds"


FEEDS
=====

An import and aggregation framework for Drupal.
http://drupal.org/project/feeds

Features
========

- Pluggable import configurations consisting of fetchers (get data) parsers
  (read and transform data) and processors (create content on Drupal).
-- HTTP upload (with optional PubSubHubbub support).
-- File upload.
-- CSV, RSS, Atom parsing.
-- Creates nodes or terms.
-- Creates lightweight database records if Data module is installed.
   http://drupal.org/project/data
-- Additional fetchers/parsers or processors can be added by an object oriented
   plugin system.
-- Granular mapping of parsed data to content elements.
- Import configurations can be piggy backed on nodes (thus using nodes to track
  subscriptions to feeds) or they can be used on a standalone form.
- Unlimited number of import configurations.
- Export import configurations to code.
- Optional libraries module support.

Requirements
============

- CTools 7.x-1.x
  http://drupal.org/project/ctools
- Job Scheduler
  http://drupal.org/project/job_scheduler
- Drupal 7.x
  http://drupal.org/project/drupal
- PHP safe mode is not supported, depending on your Feeds Importer configuration
  safe mode may cause no problems though.

Installation
============

- Install Feeds, Feeds Admin UI.
- To get started quick, install one or all of the following Feature modules:
  Feeds News, Feeds Import, Feeds Fast News (more info below).
- Make sure cron is correctly configured http://drupal.org/cron
- Go to import/ to import data.

SimplePie Installation
======================

- To install the SimplePie parser plugin, complete the following steps:
  1. Download SimplePie from http://simplepie.org/downloads. The recommended
     version is: 1.3.
  2. Decompress the downloaded zip file.
  3. Rename the uncompressed folder to 'simplepie'.
     For example rename 'simplepie-simplepie-e9472a1' to 'simplepie'.
  4. Move the folder to sites/all/libraries. The final directory structure
     should be sites/all/libraries/simplepie.
  5. Flush the Drupal cache.
  6. The SimplePie parser should be available now in the list of parsers.

Feature modules
===============

Feeds ships with three feature modules that can be enabled on
admin/build/modules or - if you are using Features - on admin/build/features.
http://drupal.org/project/features

The purpose of these modules is to provide a quick start for using Feeds. You
can either use them out of the box as they come or you can take them as samples
to learn how to build import or aggregation functionality with Feeds.

The feature modules merely contain sets of configurations using Feeds and in
some cases the modules Node, Views or Data. If the default configurations do not
fit your use case you can change them on the respective configuration pages for
Feeds, Node, Views or Data.

Here is a description of the provided feature modules:

- Feeds News -

This feature is a news aggregator. It provides a content type "Feed" that can
be used to subscribe to RSS or Atom feeds. Every item on such a feed is
aggregated as a node of the type "Feed item", also provided by the module.

What's neat about Feeds News is that it comes with a configured View that shows
a list of news items with every feed on the feed node's "View items" tab. It
also comes with an OPML importer filter that can be accessed under /import.

- Feeds Import -

This feature is an example illustrating Feeds' import capabilities. It contains
a node importer and a user importer that can be accessed under /import. Both
accept CSV or TSV files as imports.

PubSubHubbub support
====================

Feeds supports the PubSubHubbub publish/subscribe protocol. Follow these steps
to set it up for your site.
https://github.com/pubsubhubbub/

- Go to admin/build/feeds and edit (override) the importer configuration you
  would like to use for PubSubHubbub.
- Choose the HTTP Fetcher if it is not already selected.
- On the HTTP Fetcher, click on 'settings' and check "Use PubSubHubbub".
- Optionally you can use a designated hub such as http://superfeedr.com/ or your
  own. If a designated hub is specified, every feed on this importer
  configuration will be subscribed to this hub, no matter what the feed itself
  specifies.

Libraries support
=================

If you are using Libraries module, you can place external libraries in the
Libraries module's search path (for instance sites/all/libraries. The only
external library used at the moment is SimplePie.

Libraries found in the libraries search path are preferred over libraries in
feeds/libraries/.

Transliteration support
=======================

If you plan to store files with Feeds - for instance when storing podcasts
or images from syndication feeds - it is recommended to enable the
Transliteration module to avoid issues with non-ASCII characters in file names.
http://drupal.org/project/transliteration

API Overview
============

See "The developer's guide to Feeds":
http://drupal.org/node/622700

Running the Drush integration tests
===================================

In order the run Drush integration tests, Drush itself needs to be installed
with its *dev dependencies*. Furthermore, the phpunit version that comes with
Drush should be used for running the tests (instead of a globally installed
phpunit), as that one has proven to be compatible with the Drush tests.

  1. Git clone of Drush 8.

       git clone --branch 8.x https://github.com/drush-ops/drush.git
       cd drush

  2. Install Drush with dev dependencies using Composer.

       composer install

     And ensure that the following text is displayed:

       "Loading composer repositories with package information
       Installing dependencies (including require-dev) from lock file"

     Especially note that Composer says 'including require-dev'. This means that
     the Drush dev dependencies are installed (including phpunit).

  3. Execute a command like the following:

       UNISH_NO_TIMEOUTS=1 UNISH_DRUPAL_MAJOR_VERSION=7 /path/to/drush/vendor/bin/phpunit --configuration /path/to/drush/tests /path/to/feeds/tests/drush

     Replace '/path/to' with the appropriate path to the directory in question.
     Also be sure to point to the phpunit version that comes with Drush.

     So if Drush is installed in /users/megachriz/drush and the Feeds module is
     located at /users/megachriz/Sites/drupal7/sites/all/modules/feeds:

       UNISH_NO_TIMEOUTS=1 UNISH_DRUPAL_MAJOR_VERSION=7 /users/megachriz/drush/vendor/bin/phpunit --configuration /users/megachriz/drush/tests /users/megachriz/Sites/drupal7/sites/all/modules/feeds/tests/drush

Debugging
=========

Set the Drupal variable 'feeds_debug' to TRUE (i. e. using drush). This will
create a file /tmp/feeds_[my_site_location].log. Use "tail -f" on the command
line to get a live view of debug output.

Note: at the moment, only PubSubHubbub related actions are logged.

Performance
===========

See "The site builder's guide to Feeds":
http://drupal.org/node/622698

Hidden settings
===============

Hidden settings are variables that you can define by adding them to the $conf
array in your settings.php file.

Name:        feeds_debug
Default:     FALSE
Description: Set to TRUE for enabling debug output to
             /DRUPALTMPDIR/feeds_[sitename].log

Name:        feeds_library_dir
Default:     FALSE
Description: The location where Feeds should look for libraries that it uses.
             You can use this variable to override the libraries that are in
             the Feeds libraries folder, for example "http_request.inc".

Name:        feeds_importer_class
Default:     'FeedsImporter'
Description: The class to use for importing feeds.

Name:        feeds_source_class
Default:     'FeedsSource'
Description: The class to use for handling feed sources.

Name:        feeds_process_limit
Default:     50
             The number of nodes feed node processor creates or deletes in one
             page load.

Name:        http_request_timeout
Default:     15
Description: Timeout in seconds to wait for an HTTP get request to finish.
Note:        This setting could be overridden per importer in admin UI :
             admin/structure/feeds/<your_importer>/settings/<your_fetcher> page.

Name:        feeds_never_use_curl
Default:     FALSE
Description: Flag to stop feeds from using its cURL for http requests. See
             http_request_use_curl().

Name:        feeds_http_file_cache_dir
Default:     private://feeds/cache
Description: The location on the file system where results of HTTP requests are
             cached.

Name:        feeds_in_progress_dir
Default:     private://feeds/in_progress
Description: The location on the file system where temporary files are stored
             that are in progress of being imported.

Name:        feeds_sync_cache_feeds_http_interval
Default:     21600
Description: How often the feeds cache directory should be checked for orphaned
             cache files.

Name:        feeds_use_mbstring
Default:     TRUE
Description: The extension mbstring is used to convert encodings during parsing.
             The reason that this can be turned off is to be able to test Feeds
             behavior when the extension is not available.

Glossary
========

See "Feeds glossary":
http://drupal.org/node/622710

File

README.txt
View source
  1. "It feeds"
  2. FEEDS
  3. =====
  4. An import and aggregation framework for Drupal.
  5. http://drupal.org/project/feeds
  6. Features
  7. ========
  8. - Pluggable import configurations consisting of fetchers (get data) parsers
  9. (read and transform data) and processors (create content on Drupal).
  10. -- HTTP upload (with optional PubSubHubbub support).
  11. -- File upload.
  12. -- CSV, RSS, Atom parsing.
  13. -- Creates nodes or terms.
  14. -- Creates lightweight database records if Data module is installed.
  15. http://drupal.org/project/data
  16. -- Additional fetchers/parsers or processors can be added by an object oriented
  17. plugin system.
  18. -- Granular mapping of parsed data to content elements.
  19. - Import configurations can be piggy backed on nodes (thus using nodes to track
  20. subscriptions to feeds) or they can be used on a standalone form.
  21. - Unlimited number of import configurations.
  22. - Export import configurations to code.
  23. - Optional libraries module support.
  24. Requirements
  25. ============
  26. - CTools 7.x-1.x
  27. http://drupal.org/project/ctools
  28. - Job Scheduler
  29. http://drupal.org/project/job_scheduler
  30. - Drupal 7.x
  31. http://drupal.org/project/drupal
  32. - PHP safe mode is not supported, depending on your Feeds Importer configuration
  33. safe mode may cause no problems though.
  34. Installation
  35. ============
  36. - Install Feeds, Feeds Admin UI.
  37. - To get started quick, install one or all of the following Feature modules:
  38. Feeds News, Feeds Import, Feeds Fast News (more info below).
  39. - Make sure cron is correctly configured http://drupal.org/cron
  40. - Go to import/ to import data.
  41. SimplePie Installation
  42. ======================
  43. - To install the SimplePie parser plugin, complete the following steps:
  44. 1. Download SimplePie from http://simplepie.org/downloads. The recommended
  45. version is: 1.3.
  46. 2. Decompress the downloaded zip file.
  47. 3. Rename the uncompressed folder to 'simplepie'.
  48. For example rename 'simplepie-simplepie-e9472a1' to 'simplepie'.
  49. 4. Move the folder to sites/all/libraries. The final directory structure
  50. should be sites/all/libraries/simplepie.
  51. 5. Flush the Drupal cache.
  52. 6. The SimplePie parser should be available now in the list of parsers.
  53. Feature modules
  54. ===============
  55. Feeds ships with three feature modules that can be enabled on
  56. admin/build/modules or - if you are using Features - on admin/build/features.
  57. http://drupal.org/project/features
  58. The purpose of these modules is to provide a quick start for using Feeds. You
  59. can either use them out of the box as they come or you can take them as samples
  60. to learn how to build import or aggregation functionality with Feeds.
  61. The feature modules merely contain sets of configurations using Feeds and in
  62. some cases the modules Node, Views or Data. If the default configurations do not
  63. fit your use case you can change them on the respective configuration pages for
  64. Feeds, Node, Views or Data.
  65. Here is a description of the provided feature modules:
  66. - Feeds News -
  67. This feature is a news aggregator. It provides a content type "Feed" that can
  68. be used to subscribe to RSS or Atom feeds. Every item on such a feed is
  69. aggregated as a node of the type "Feed item", also provided by the module.
  70. What's neat about Feeds News is that it comes with a configured View that shows
  71. a list of news items with every feed on the feed node's "View items" tab. It
  72. also comes with an OPML importer filter that can be accessed under /import.
  73. - Feeds Import -
  74. This feature is an example illustrating Feeds' import capabilities. It contains
  75. a node importer and a user importer that can be accessed under /import. Both
  76. accept CSV or TSV files as imports.
  77. PubSubHubbub support
  78. ====================
  79. Feeds supports the PubSubHubbub publish/subscribe protocol. Follow these steps
  80. to set it up for your site.
  81. https://github.com/pubsubhubbub/
  82. - Go to admin/build/feeds and edit (override) the importer configuration you
  83. would like to use for PubSubHubbub.
  84. - Choose the HTTP Fetcher if it is not already selected.
  85. - On the HTTP Fetcher, click on 'settings' and check "Use PubSubHubbub".
  86. - Optionally you can use a designated hub such as http://superfeedr.com/ or your
  87. own. If a designated hub is specified, every feed on this importer
  88. configuration will be subscribed to this hub, no matter what the feed itself
  89. specifies.
  90. Libraries support
  91. =================
  92. If you are using Libraries module, you can place external libraries in the
  93. Libraries module's search path (for instance sites/all/libraries. The only
  94. external library used at the moment is SimplePie.
  95. Libraries found in the libraries search path are preferred over libraries in
  96. feeds/libraries/.
  97. Transliteration support
  98. =======================
  99. If you plan to store files with Feeds - for instance when storing podcasts
  100. or images from syndication feeds - it is recommended to enable the
  101. Transliteration module to avoid issues with non-ASCII characters in file names.
  102. http://drupal.org/project/transliteration
  103. API Overview
  104. ============
  105. See "The developer's guide to Feeds":
  106. http://drupal.org/node/622700
  107. Running the Drush integration tests
  108. ===================================
  109. In order the run Drush integration tests, Drush itself needs to be installed
  110. with its *dev dependencies*. Furthermore, the phpunit version that comes with
  111. Drush should be used for running the tests (instead of a globally installed
  112. phpunit), as that one has proven to be compatible with the Drush tests.
  113. 1. Git clone of Drush 8.
  114. git clone --branch 8.x https://github.com/drush-ops/drush.git
  115. cd drush
  116. 2. Install Drush with dev dependencies using Composer.
  117. composer install
  118. And ensure that the following text is displayed:
  119. "Loading composer repositories with package information
  120. Installing dependencies (including require-dev) from lock file"
  121. Especially note that Composer says 'including require-dev'. This means that
  122. the Drush dev dependencies are installed (including phpunit).
  123. 3. Execute a command like the following:
  124. UNISH_NO_TIMEOUTS=1 UNISH_DRUPAL_MAJOR_VERSION=7 /path/to/drush/vendor/bin/phpunit --configuration /path/to/drush/tests /path/to/feeds/tests/drush
  125. Replace '/path/to' with the appropriate path to the directory in question.
  126. Also be sure to point to the phpunit version that comes with Drush.
  127. So if Drush is installed in /users/megachriz/drush and the Feeds module is
  128. located at /users/megachriz/Sites/drupal7/sites/all/modules/feeds:
  129. UNISH_NO_TIMEOUTS=1 UNISH_DRUPAL_MAJOR_VERSION=7 /users/megachriz/drush/vendor/bin/phpunit --configuration /users/megachriz/drush/tests /users/megachriz/Sites/drupal7/sites/all/modules/feeds/tests/drush
  130. Debugging
  131. =========
  132. Set the Drupal variable 'feeds_debug' to TRUE (i. e. using drush). This will
  133. create a file /tmp/feeds_[my_site_location].log. Use "tail -f" on the command
  134. line to get a live view of debug output.
  135. Note: at the moment, only PubSubHubbub related actions are logged.
  136. Performance
  137. ===========
  138. See "The site builder's guide to Feeds":
  139. http://drupal.org/node/622698
  140. Hidden settings
  141. ===============
  142. Hidden settings are variables that you can define by adding them to the $conf
  143. array in your settings.php file.
  144. Name: feeds_debug
  145. Default: FALSE
  146. Description: Set to TRUE for enabling debug output to
  147. /DRUPALTMPDIR/feeds_[sitename].log
  148. Name: feeds_library_dir
  149. Default: FALSE
  150. Description: The location where Feeds should look for libraries that it uses.
  151. You can use this variable to override the libraries that are in
  152. the Feeds libraries folder, for example "http_request.inc".
  153. Name: feeds_importer_class
  154. Default: 'FeedsImporter'
  155. Description: The class to use for importing feeds.
  156. Name: feeds_source_class
  157. Default: 'FeedsSource'
  158. Description: The class to use for handling feed sources.
  159. Name: feeds_process_limit
  160. Default: 50
  161. The number of nodes feed node processor creates or deletes in one
  162. page load.
  163. Name: http_request_timeout
  164. Default: 15
  165. Description: Timeout in seconds to wait for an HTTP get request to finish.
  166. Note: This setting could be overridden per importer in admin UI :
  167. admin/structure/feeds//settings/ page.
  168. Name: feeds_never_use_curl
  169. Default: FALSE
  170. Description: Flag to stop feeds from using its cURL for http requests. See
  171. http_request_use_curl().
  172. Name: feeds_http_file_cache_dir
  173. Default: private://feeds/cache
  174. Description: The location on the file system where results of HTTP requests are
  175. cached.
  176. Name: feeds_in_progress_dir
  177. Default: private://feeds/in_progress
  178. Description: The location on the file system where temporary files are stored
  179. that are in progress of being imported.
  180. Name: feeds_sync_cache_feeds_http_interval
  181. Default: 21600
  182. Description: How often the feeds cache directory should be checked for orphaned
  183. cache files.
  184. Name: feeds_use_mbstring
  185. Default: TRUE
  186. Description: The extension mbstring is used to convert encodings during parsing.
  187. The reason that this can be turned off is to be able to test Feeds
  188. behavior when the extension is not available.
  189. Glossary
  190. ========
  191. See "Feeds glossary":
  192. http://drupal.org/node/622710