You are here

migrate.module in Migrate 7.2

Same filename and directory in other branches
  1. 6.2 migrate.module
  2. 6 migrate.module

API and drush commands to support migration of data from external sources into a Drupal installation.

File

migrate.module
View source
<?php

/**
 * @file
 * API and drush commands to support migration of data from external sources
 * into a Drupal installation.
 */

// TODO:
// Continue hook_schema_alter() for map & message tables?
// Views hooks for map/message tables
// xlat support?
// Documentation
// Tests
define('MIGRATE_API_VERSION', 2);
define('MIGRATE_ACCESS_BASIC', 'migration information');
define('MIGRATE_ACCESS_ADVANCED', 'advanced migration information');

/**
 * Implements hook_help().
 */
function migrate_help($path, $arg) {
  switch ($path) {
    case 'admin/help#migrate':
      $output = '';
      $output .= '<h3>' . t('About') . '</h3>';
      $output .= '<p>' . t('The Migrate module provides a flexible framework for migrating content into Drupal from other sources (e.g., when converting a web site from another CMS to Drupal). Out-of-the-box, support for creating Drupal nodes, taxonomy terms, comments, and users are included. Plugins permit migration of other types of content. For more information, see the online documentation for the <a href="@handbook">Migrate module</a>.', array(
        '@handbook' => 'http://drupal.org/migrate',
      )) . '</p>';
      $output .= '<h3>' . t('Uses') . '</h3>';
      $output .= '<dl>';
      $output .= '<dt>' . t('Getting Started') . '</dt>';
      $output .= '<dd>' . t('To get started, enable the migrate_example module and browse to admin/content/migrate to see its dashboard. The code for this migration is in migrate_example/beer.inc (advanced examples are in wine.inc). Mimic that file in order to specify your own migrations.') . '</dd>';
      $output .= '<dt>' . t('Migrate Support') . '</dt>';
      $output .= '<dd>' . t('The Migrate module itself has support for migration into core objects. Some built-in support for migration involving contrib modules is in the migrate_extras module.') . '</dd>';
      $output .= '<dt>' . t('Migrate Extended Support') . '</dt>';
      $output .= '<dd>' . t('The best place to implement migration support for a contributed module is in that module, not in the Migrate or Migrate Extras modules. That way, the migration support is always self-consistent with the current module implementation - it is not practical for the migrate modules to keep up with changes to all other contrib modules.') . '</dd>';
      $output .= '</dl>';
      return $output;
  }
}

/**
 * Retrieve a list of all active migrations, ordered by dependencies. To be
 * recognized, a class must be non-abstract, and derived from MigrationBase.
 *
 * @param $reset
 *  If TRUE, the static cache of migrations will be flushed before attempting to
 *  reinstantiate all active migrations. This can be important for script runs
 *  where migration classes may be dynamically registered.
 *
 * @return
 *  Array of migration objects, keyed by the machine name.
 */
function migrate_migrations($reset = NULL) {
  static $migrations = array();
  if (!empty($migrations) && empty($reset)) {
    return $migrations;
  }

  // Get list of modules implementing Migrate API - mainly, we're looking to
  // make sure any dynamic migrations defined in hook_migrate_api() get registered.
  migrate_get_module_apis(TRUE);
  $dependencies_list = array();
  $dependent_migrations = array();
  $required_migrations = array();
  $result = db_select('migrate_status', 'ms')
    ->fields('ms', array(
    'machine_name',
    'class_name',
  ))
    ->execute();
  foreach ($result as $row) {
    if (class_exists($row->class_name)) {
      $reflect = new ReflectionClass($row->class_name);
      if (!$reflect
        ->isAbstract() && $reflect
        ->isSubclassOf('MigrationBase')) {
        $migration = MigrationBase::getInstance($row->machine_name);
        if ($migration) {
          $dependencies = $migration
            ->getDependencies();
          $dependencies_list[$row->machine_name] = $dependencies;
          if (count($dependencies) > 0) {

            // Set classes with dependencies aside for reordering
            $dependent_migrations[$row->machine_name] = $migration;
            $required_migrations += $dependencies;
          }
          else {

            // No dependencies, just add
            $migrations[$row->machine_name] = $migration;
          }
        }
      }
      else {
        MigrationBase::displayMessage(t('Class !class is no longer a valid concrete migration class', array(
          '!class' => $row->class_name,
        )));
      }
    }
    else {
      MigrationBase::displayMessage(t('Class !class could not be found', array(
        '!class' => $row->class_name,
      )));
    }
  }
  $ordered_migrations = migrate_order_dependencies($dependencies_list);
  foreach ($ordered_migrations as $name) {
    if (!isset($migrations[$name])) {
      $migrations[$name] = $dependent_migrations[$name];
    }
  }

  // The migrations are now ordered according to their own dependencies - now order
  // them by group
  $groups = MigrateGroup::groups();

  // Seed the final list by properly-ordered groups.
  $final_migrations = array();
  foreach ($groups as $name => $group) {
    $final_migrations[$name] = array();
  }

  // Fill in the grouped list.
  foreach ($migrations as $machine_name => $migration) {
    if (!method_exists($migration, 'getGroup')) {
      MigrationBase::displayMessage(t('Migration !machine_name is not a valid Migration dependency.', array(
        '!machine_name' => $machine_name,
      )));
    }
    else {
      $final_migrations[$migration
        ->getGroup()
        ->getName()][$machine_name] = $migration;
    }
  }

  // Flatten the grouped list.
  $migrations = array();
  foreach ($final_migrations as $group_name => $group_migrations) {
    foreach ($group_migrations as $machine_name => $migration) {
      $migrations[$machine_name] = $migration;
    }
  }
  return $migrations;
}

/**
 * Invoke any available handlers attached to a given destination type.
 * If any handlers have dependencies defined, they will be invoked after
 * the specified handlers.
 *
 * @param $destination
 *  Destination type ('Node', 'User', etc.) - generally the same string as
 *  the destination class name without the MigrateDestination prefix.
 * @param $method
 *  Method name such as 'prepare' (called at the beginning of an import
 *   operation) or 'complete' (called at the end of an import operation).
 * @param ...
 *  Parameters to be passed to the handler.
 */
function migrate_handler_invoke_all($destination, $method) {
  $args = func_get_args();
  array_shift($args);
  array_shift($args);
  $return = array();
  $class_list = _migrate_class_list('MigrateDestinationHandler');
  $disabled = unserialize(variable_get('migrate_disabled_handlers', serialize(array())));
  foreach ($class_list as $class_name => $handler) {
    if (!in_array($class_name, $disabled) && $handler
      ->handlesType($destination) && method_exists($handler, $method)) {
      migrate_instrument_start($class_name . '->' . $method);
      $result = call_user_func_array(array(
        $handler,
        $method,
      ), $args);
      migrate_instrument_stop($class_name . '->' . $method);
      if (isset($result) && is_array($result)) {
        $return = array_merge($return, $result);
      }
      elseif (isset($result)) {
        $return[] = $result;
      }
    }
  }
  return $return;
}

/**
 * Invoke any available handlers attached to a given field type.
 *
 * @param $entity
 *  The object we are building up before calling example_save().
 * @param $field_info
 *  Array of info on the field, from field_info_field().
 * @param $instance
 *  Array of info in the field instance, from field_info_instances().
 * @param $values
 *  Array of incoming values, to be transformed into the appropriate structure
 *  for the field type.
 * @param $method
 *  Handler method to call (defaults to prepare()).
 */
function migrate_field_handler_invoke_all($entity, array $field_info, array $instance, array $values, $method = 'prepare') {
  static $types_handled = array();
  static $methods_handled = array();
  static $disabled = null;
  $return = array();
  $type = $field_info['type'];
  static $class_list = null;
  if (!$class_list) {
    $class_list = _migrate_class_list('MigrateFieldHandler');
  }

  // No need to do this unserialize/variable_get/serialize so often,
  // it never changes.
  if (!is_array($disabled)) {
    $disabled = unserialize(variable_get('migrate_disabled_handlers', serialize(array())));
  }

  // This function is called a lot. Rather than determine if the field type is
  // handled once for every record, the value should be determined once per
  // execution.
  // The same can go for whether the handler/method pair exists
  if (!isset($types_handled[$type])) {
    foreach ($class_list as $class_name => $handler) {
      $types_handled[$type][$class_name] = $handler
        ->handlesType($type);
    }
  }
  if (!isset($methods_handled[$method])) {
    foreach ($class_list as $class_name => $handler) {
      $methods_handled[$method][$class_name] = method_exists($handler, $method);
    }
  }
  $handler_called = FALSE;
  foreach ($class_list as $class_name => $handler) {
    if (!in_array($class_name, $disabled) && $types_handled[$type][$class_name] && $methods_handled[$method][$class_name]) {
      migrate_instrument_start($class_name . '->' . $method);
      $result = call_user_func_array(array(
        $handler,
        $method,
      ), array(
        $entity,
        $field_info,
        $instance,
        $values,
      ));
      $handler_called = TRUE;
      migrate_instrument_stop($class_name . '->' . $method);
      if (isset($result) && is_array($result)) {
        $return = array_merge_recursive($return, $result);
      }
      elseif (isset($result)) {
        $return[] = $result;
      }
    }
  }
  if (!$handler_called && $method == 'prepare') {
    $handler = new MigrateDefaultFieldHandler();
    migrate_instrument_start('MigrateDefaultFieldHandler->prepare');
    $result = call_user_func_array(array(
      $handler,
      'prepare',
    ), array(
      $entity,
      $field_info,
      $instance,
      $values,
    ));
    migrate_instrument_stop('MigrateDefaultFieldHandler->prepare');
    if (isset($result) && is_array($result)) {
      $return = array_merge_recursive($return, $result);
    }
    elseif (isset($result)) {
      $return[] = $result;
    }
  }
  return $return;
}

/**
 * For a given parent class, identify and instantiate objects for any
 * non-abstract classes derived from the parent, returning an array of the
 * objects indexed by class name. The array will be ordered such that any
 * classes with dependencies are listed after the classes they are dependent
 * on.
 *
 * @param $parent_class
 *  Name of a class from which results will be derived.
 *
 * @return
 *  Array of objects, keyed by the class name.
 */
function _migrate_class_list($parent_class) {

  // Get info on modules implementing Migrate API
  static $module_info;
  if (!isset($module_info)) {
    $module_info = migrate_get_module_apis();
  }
  static $class_lists = array();
  if (!isset($class_lists[$parent_class])) {
    $class_lists[$parent_class] = array();
    if ($parent_class == 'MigrateDestinationHandler') {
      $handler_key = 'destination handlers';
    }
    else {
      $handler_key = 'field handlers';
    }

    // Add explicitly-registered handler classes
    foreach ($module_info as $info) {
      if (isset($info[$handler_key]) && is_array($info[$handler_key])) {
        foreach ($info[$handler_key] as $handler_class) {
          $class_lists[$parent_class][$handler_class] = new $handler_class();
        }
      }
    }
  }
  return $class_lists[$parent_class];
}

/**
 * Implements hook_hook_info().
 */
function migrate_hook_info() {

  // Look for hook_migrate_api() in example.migrate.inc.
  $hooks['migrate_api'] = array(
    'group' => 'migrate',
  );
  $hooks['migrate_api_alter'] = array(
    'group' => 'migrate',
  );
  return $hooks;
}

/**
 * Implementation of hook_permission().
 */
function migrate_permission() {
  return array(
    MIGRATE_ACCESS_BASIC => array(
      'title' => t('Access to basic migration information'),
    ),
    MIGRATE_ACCESS_ADVANCED => array(
      'title' => t('Access to advanced migration information'),
    ),
  );
}

/**
 * Get a list of modules that support the current migrate API.
 */
function migrate_get_module_apis($reset = FALSE) {
  static $cache = NULL;
  if ($reset) {
    $cache = NULL;
  }
  if (!isset($cache)) {
    $cache = array();
    foreach (module_implements('migrate_api') as $module) {
      $function = $module . '_migrate_api';
      $info = $function();
      if (isset($info['api']) && $info['api'] == MIGRATE_API_VERSION) {
        $cache[$module] = $info;
      }
      else {
        drupal_set_message(t('%function supports Migrate API version %modversion,
          Migrate module API version is %version - migration support not loaded.', array(
          '%function' => $function,
          '%modversion' => $info['api'],
          '%version' => MIGRATE_API_VERSION,
        )));
      }
    }

    // Allow modules to alter the migration information.
    drupal_alter('migrate_api', $cache);
  }
  return $cache;
}

/**
 * Register any migrations defined in hook_migrate_api().
 *
 * @param array $machine_names
 *  If populated, only (re)register the specified migrations.
 */
function migrate_static_registration($machine_names = array()) {
  $module_info = migrate_get_module_apis(TRUE);
  foreach ($module_info as $module => $info) {

    // Register any groups defined via the hook.
    if (isset($info['groups']) && is_array($info['groups'])) {
      foreach ($info['groups'] as $name => $arguments) {
        $title = $arguments['title'];
        unset($arguments['title']);
        MigrateGroup::register($name, $title, $arguments);
      }
    }

    // Register any migrations defined via the hook.
    if (isset($info['migrations']) && is_array($info['migrations'])) {
      foreach ($info['migrations'] as $machine_name => $arguments) {

        // If we have an explicit list to register, skip any not in the list.
        if (!empty($machine_names) && !in_array($machine_name, $machine_names)) {
          continue;
        }
        $class_name = $arguments['class_name'];
        unset($arguments['class_name']);

        // Call the right registerMigration implementation. Note that this means
        // that classes that override registerMigration() must always call it
        // directly, they cannot register those classes by defining them in
        // hook_migrate_api() and expect their extension to be called.
        if (is_subclass_of($class_name, 'Migration')) {
          Migration::registerMigration($class_name, $machine_name, $arguments);
        }
        else {
          MigrationBase::registerMigration($class_name, $machine_name, $arguments);
        }
      }
    }
  }
}

/**
 * Do a topological sort on our dependencies graph.
 */
function migrate_order_dependencies($dependencies) {
  $visited = array();
  $list = array();
  foreach (array_keys($dependencies) as $name) {
    $visited[$name] = FALSE;
  }
  foreach (array_keys($dependencies) as $name) {
    migrate_visit_dependent($dependencies, $name, $list, $visited);
  }
  return $list;
}

/**
 * Depth-first search for independent migrations.
 */
function migrate_visit_dependent($dependencies, $name, &$list, &$visited) {
  if ($visited[$name]) {
    if ($list[$name]) {
      return;
    }
    else {
      throw new MigrateException(t('Failure to sort migration list due to circular dependencies involving %name.', array(
        '%name' => $name,
      )));
    }
  }
  $visited[$name] = TRUE;
  if (isset($dependencies[$name])) {
    foreach ($dependencies[$name] as $dependent) {
      migrate_visit_dependent($dependencies, $dependent, $list, $visited);
    }
  }
  $list[$name] = $name;
}

/**
 * Implements hook_watchdog().
 * Find the migration that is currently running and notify it.
 *
 * @param array $log_entry
 */
function migrate_watchdog($log_entry) {

  // Ensure that the Migration class exists, as different bootstrap phases may
  // not have included migration.inc yet.
  if (class_exists('Migration') && ($migration = Migration::currentMigration())) {
    switch ($log_entry['severity']) {
      case WATCHDOG_EMERGENCY:
      case WATCHDOG_ALERT:
      case WATCHDOG_CRITICAL:
      case WATCHDOG_ERROR:
        $severity = MigrationBase::MESSAGE_ERROR;
        break;
      case WATCHDOG_WARNING:
        $severity = MigrationBase::MESSAGE_WARNING;
        break;
      case WATCHDOG_NOTICE:
        $severity = MigrationBase::MESSAGE_NOTICE;
        break;
      case WATCHDOG_DEBUG:
      case WATCHDOG_INFO:
      default:
        $severity = MigrationBase::MESSAGE_INFORMATIONAL;
        break;
    }
    $variables = is_array($log_entry['variables']) ? $log_entry['variables'] : array();
    $migration
      ->saveMessage(t($log_entry['message'], $variables), $severity);
  }
}

/**
 * Resource functions modeled on Drupal's timer functions
 */

/**
 * Save memory usage with the specified name. If you start and stop the same
 * memory name multiple times, the measured differences will be accumulated.
 *
 * @param name
 *   The name of the memory measurement.
 */
function migrate_memory_start($name) {
  global $_migrate_memory;
  $_migrate_memory[$name]['start'] = memory_get_usage();
  $_migrate_memory[$name]['count'] = isset($_migrate_memory[$name]['count']) ? ++$_migrate_memory[$name]['count'] : 1;
}

/**
 * Read the current memory value without recording the change.
 *
 * @param name
 *   The name of the memory measurement.
 *
 * @return
 *   The change in bytes since the last start.
 */
function migrate_memory_read($name) {
  global $_migrate_memory;
  if (isset($_migrate_memory[$name]['start'])) {
    $stop = memory_get_usage();
    $diff = $stop - $_migrate_memory[$name]['start'];
    if (isset($_migrate_memory[$name]['bytes'])) {
      $diff += $_migrate_memory[$name]['bytes'];
    }
    return $diff;
  }
  return $_migrate_memory[$name]['bytes'];
}

/**
 * Stop the memory counter with the specified name.
 *
 * @param name
 *   The name of the memory measurement.
 *
 * @return
 *   A memory array. The array contains the number of times the memory has been
 *   started and stopped (count) and the accumulated memory difference value in
 *   bytes.
 */
function migrate_memory_stop($name) {
  global $_migrate_memory;
  if (isset($_migrate_memory[$name])) {
    if (isset($_migrate_memory[$name]['start'])) {
      $stop = memory_get_usage();
      $diff = $stop - $_migrate_memory[$name]['start'];
      if (isset($_migrate_memory[$name]['bytes'])) {
        $_migrate_memory[$name]['bytes'] += $diff;
      }
      else {
        $_migrate_memory[$name]['bytes'] = $diff;
      }
      unset($_migrate_memory[$name]['start']);
    }
    return $_migrate_memory[$name];
  }
}

/**
 * Start measuring time and (optionally) memory consumption over a section of
 * code. Note that the memory consumption measurement is generally not useful
 * in lower areas of the code, where data is being generated that will be freed
 * by the next call to the same area. For example, measuring the memory
 * consumption of db_query is not going to be helpful.
 *
 * @param $name
 *  The name of the measurement.
 * @param $include_memory
 *  Measure both memory and timers. Defaults to FALSE (timers only).
 */
function migrate_instrument_start($name, $include_memory = FALSE) {
  global $_migrate_track_memory, $_migrate_track_timer;
  if ($_migrate_track_memory && $include_memory) {
    migrate_memory_start($name);
  }
  if ($_migrate_track_timer) {
    timer_start($name);
  }
}

/**
 * Stop measuring both memory and time consumption over a section of code.
 *
 * @param $name
 *  The name of the measurement.
 */
function migrate_instrument_stop($name) {
  global $_migrate_track_memory, $_migrate_track_timer;
  if ($_migrate_track_timer) {
    timer_stop($name);
  }
  if ($_migrate_track_memory) {
    migrate_memory_stop($name);
  }
}

/**
 * Call hook_migrate_overview for overall documentation on implemented
 * migrations.
 */
function migrate_overview() {
  $overview = '';
  $results = module_invoke_all('migrate_overview');
  foreach ($results as $result) {
    $overview .= $result . ' ';
  }
  return $overview;
}

/**
 * Implements hook_modules_enabled.
 */
function migrate_modules_enabled($modules) {
  if (array_intersect($modules, module_implements('migrate_api'))) {
    migrate_static_registration();
  }
}

/**
 * Implements hook_module_implements_alter().
 */
function migrate_module_implements_alter(&$implementation, $hook) {

  // Ensure that the Migration class exists, as different bootstrap phases may
  // not have included migration.inc yet.
  if (class_exists('Migration') && ($migration = Migration::currentMigration())) {
    $disable_hooks = $migration
      ->getDisableHooks();
    if (isset($disable_hooks[$hook])) {
      foreach ($disable_hooks[$hook] as $module) {
        unset($implementation[$module]);
      }
    }
  }
}

Functions

Namesort descending Description
migrate_field_handler_invoke_all Invoke any available handlers attached to a given field type.
migrate_get_module_apis Get a list of modules that support the current migrate API.
migrate_handler_invoke_all Invoke any available handlers attached to a given destination type. If any handlers have dependencies defined, they will be invoked after the specified handlers.
migrate_help Implements hook_help().
migrate_hook_info Implements hook_hook_info().
migrate_instrument_start Start measuring time and (optionally) memory consumption over a section of code. Note that the memory consumption measurement is generally not useful in lower areas of the code, where data is being generated that will be freed by the next call to the…
migrate_instrument_stop Stop measuring both memory and time consumption over a section of code.
migrate_memory_read Read the current memory value without recording the change.
migrate_memory_start Save memory usage with the specified name. If you start and stop the same memory name multiple times, the measured differences will be accumulated.
migrate_memory_stop Stop the memory counter with the specified name.
migrate_migrations Retrieve a list of all active migrations, ordered by dependencies. To be recognized, a class must be non-abstract, and derived from MigrationBase.
migrate_modules_enabled Implements hook_modules_enabled.
migrate_module_implements_alter Implements hook_module_implements_alter().
migrate_order_dependencies Do a topological sort on our dependencies graph.
migrate_overview Call hook_migrate_overview for overall documentation on implemented migrations.
migrate_permission Implementation of hook_permission().
migrate_static_registration Register any migrations defined in hook_migrate_api().
migrate_visit_dependent Depth-first search for independent migrations.
migrate_watchdog Implements hook_watchdog(). Find the migration that is currently running and notify it.
_migrate_class_list For a given parent class, identify and instantiate objects for any non-abstract classes derived from the parent, returning an array of the objects indexed by class name. The array will be ordered such that any classes with dependencies are listed…

Constants