You are here

Home » API reference » Migrate Tools 8.4 » MigrateBatchExecutable.php

class MigrateBatchExecutable in Migrate Tools 8.4

Same name and namespace in other branches
  1. 8.5 src/MigrateBatchExecutable.php \Drupal\migrate_tools\MigrateBatchExecutable

Defines a migrate executable class for batch migrations through UI.

Hierarchy

Expanded class hierarchy of MigrateBatchExecutable

3 files declare their use of MigrateBatchExecutable
MigrateToolsTestCommands.php in tests/modules/migrate_tools_test/src/Commands/MigrateToolsTestCommands.php
MigrationController.php in src/Controller/MigrationController.php
MigrationExecuteForm.php in src/Form/MigrationExecuteForm.php

File

src/MigrateBatchExecutable.php, line 12

Namespace

Drupal\migrate_tools
View source 
class MigrateBatchExecutable extends MigrateExecutable {

  /**
   * Representing a batch import operation.
   */
  const BATCH_IMPORT = 1;

  /**
   * Indicates if we need to update existing rows or skip them.
   *
   * @var int
   */
  protected $updateExistingRows = 0;

  /**
   * Indicates if we need import dependent migrations also.
   *
   * @var int
   */
  protected $checkDependencies = 0;

  /**
   * The current batch context.
   *
   * @var array
   */
  protected $batchContext = [];

  /**
   * Plugin manager for migration plugins.
   *
   * @var \Drupal\migrate\Plugin\MigrationPluginManagerInterface
   */
  protected $migrationPluginManager;

  /**
   * {@inheritdoc}
   */
  public function __construct(MigrationInterface $migration, MigrateMessageInterface $message, array $options = []) {
    if (isset($options['update'])) {
      $this->updateExistingRows = $options['update'];
    }
    if (isset($options['force'])) {
      $this->checkDependencies = $options['force'];
    }
    parent::__construct($migration, $message, $options);
    $this->migrationPluginManager = \Drupal::getContainer()
      ->get('plugin.manager.migration');
  }

  /**
   * Sets the current batch content so listeners can update the messages.
   *
   * @param array|\DrushBatchContext $context
   *   The batch context.
   */
  public function setBatchContext(&$context) {
    $this->batchContext =& $context;
  }

  /**
   * Gets a reference to the current batch context.
   *
   * @return array
   *   The batch context.
   */
  public function &getBatchContext() {
    return $this->batchContext;
  }

  /**
   * Setup batch operations for running the migration.
   */
  public function batchImport() {

    // Create the batch operations for each migration that needs to be executed.
    // This includes the migration for this executable, but also the dependent
    // migrations.
    $operations = $this
      ->batchOperations([
      $this->migration,
    ], 'import', [
      'limit' => $this->itemLimit,
      'update' => $this->updateExistingRows,
      'force' => $this->checkDependencies,
    ]);
    if (count($operations) > 0) {
      $batch = [
        'operations' => $operations,
        'title' => $this
          ->t('Migrating %migrate', [
          '%migrate' => $this->migration
            ->label(),
        ]),
        'init_message' => $this
          ->t('Start migrating %migrate', [
          '%migrate' => $this->migration
            ->label(),
        ]),
        'progress_message' => $this
          ->t('Migrating %migrate', [
          '%migrate' => $this->migration
            ->label(),
        ]),
        'error_message' => $this
          ->t('An error occurred while migrating %migrate.', [
          '%migrate' => $this->migration
            ->label(),
        ]),
        'finished' => '\\Drupal\\migrate_tools\\MigrateBatchExecutable::batchFinishedImport',
      ];
      batch_set($batch);
    }
  }

  /**
   * Helper to generate the batch operations for importing migrations.
   *
   * @param \Drupal\migrate\Plugin\MigrationInterface[] $migrations
   *   The migrations.
   * @param string $operation
   *   The batch operation to perform.
   * @param array $options
   *   The migration options.
   *
   * @return array
   *   The batch operations to perform.
   */
  protected function batchOperations(array $migrations, $operation, array $options = []) {
    $operations = [];
    foreach ($migrations as $id => $migration) {
      if (!empty($options['update'])) {
        $migration
          ->getIdMap()
          ->prepareUpdate();
      }
      if (!empty($options['force'])) {
        $migration
          ->set('requirements', []);
      }
      else {
        $dependencies = $migration
          ->getMigrationDependencies();
        if (!empty($dependencies['required'])) {
          $required_migrations = $this->migrationPluginManager
            ->createInstances($dependencies['required']);

          // For dependent migrations will need to be migrate all items.
          $dependent_options = $options;
          $dependent_options['limit'] = 0;
          $operations = array_merge($operations, $this
            ->batchOperations($required_migrations, $operation, [
            'limit' => 0,
            'update' => $options['update'],
            'force' => $options['force'],
          ]));
        }
      }
      $operations[] = [
        '\\Drupal\\migrate_tools\\MigrateBatchExecutable::batchProcessImport',
        [
          $migration
            ->id(),
          $options,
        ],
      ];
    }
    return $operations;
  }

  /**
   * Batch 'operation' callback.
   *
   * @param string $migration_id
   *   The migration id.
   * @param array $options
   *   The batch executable options.
   * @param array|\DrushBatchContext $context
   *   The sandbox context.
   */
  public static function batchProcessImport($migration_id, array $options, &$context) {
    if (empty($context['sandbox'])) {
      $context['finished'] = 0;
      $context['sandbox'] = [];
      $context['sandbox']['total'] = 0;
      $context['sandbox']['counter'] = 0;
      $context['sandbox']['batch_limit'] = 0;
      $context['sandbox']['operation'] = MigrateBatchExecutable::BATCH_IMPORT;
    }

    // Prepare the migration executable.
    $message = new MigrateMessage();

    /** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
    $migration = \Drupal::getContainer()
      ->get('plugin.manager.migration')
      ->createInstance($migration_id, isset($options['configuration']) ? $options['configuration'] : []);
    unset($options['configuration']);

    // Each batch run we need to reinitialize the counter for the migration.
    if (!empty($options['limit']) && isset($context['results'][$migration
      ->id()]['@numitems'])) {
      $options['limit'] = $options['limit'] - $context['results'][$migration
        ->id()]['@numitems'];
    }
    $executable = new MigrateBatchExecutable($migration, $message, $options);
    if (empty($context['sandbox']['total'])) {
      $context['sandbox']['total'] = $executable
        ->getSource()
        ->count();
      $context['sandbox']['batch_limit'] = $executable
        ->calculateBatchLimit($context);
      $context['results'][$migration
        ->id()] = [
        '@numitems' => 0,
        '@created' => 0,
        '@updated' => 0,
        '@failures' => 0,
        '@ignored' => 0,
        '@name' => $migration
          ->id(),
      ];
    }

    // Every iteration, we reset out batch counter.
    $context['sandbox']['batch_counter'] = 0;

    // Make sure we know our batch context.
    $executable
      ->setBatchContext($context);

    // Do the import.
    $result = $executable
      ->import();

    // Store the result; will need to combine the results of all our iterations.
    $context['results'][$migration
      ->id()] = [
      '@numitems' => $context['results'][$migration
        ->id()]['@numitems'] + $executable
        ->getProcessedCount(),
      '@created' => $context['results'][$migration
        ->id()]['@created'] + $executable
        ->getCreatedCount(),
      '@updated' => $context['results'][$migration
        ->id()]['@updated'] + $executable
        ->getUpdatedCount(),
      '@failures' => $context['results'][$migration
        ->id()]['@failures'] + $executable
        ->getFailedCount(),
      '@ignored' => $context['results'][$migration
        ->id()]['@ignored'] + $executable
        ->getIgnoredCount(),
      '@name' => $migration
        ->id(),
    ];

    // Do some housekeeping.
    if ($result != MigrationInterface::RESULT_INCOMPLETE) {
      $context['finished'] = 1;
    }
    else {
      $context['sandbox']['counter'] = $context['results'][$migration
        ->id()]['@numitems'];
      if ($context['sandbox']['counter'] <= $context['sandbox']['total']) {
        $context['finished'] = (double) $context['sandbox']['counter'] / (double) $context['sandbox']['total'];
        $context['message'] = t('Importing %migration (@percent%).', [
          '%migration' => $migration
            ->label(),
          '@percent' => (int) ($context['finished'] * 100),
        ]);
      }
    }
  }

  /**
   * Finished callback for import batches.
   *
   * @param bool $success
   *   A boolean indicating whether the batch has completed successfully.
   * @param array $results
   *   The value set in $context['results'] by callback_batch_operation().
   * @param array $operations
   *   If $success is FALSE, contains the operations that remained unprocessed.
   */
  public static function batchFinishedImport($success, array $results, array $operations) {
    if ($success) {
      foreach ($results as $migration_id => $result) {
        $singular_message = "Processed 1 item (@created created, @updated updated, @failures failed, @ignored ignored) - done with '@name'";
        $plural_message = "Processed @numitems items (@created created, @updated updated, @failures failed, @ignored ignored) - done with '@name'";
        \Drupal::messenger()
          ->addStatus(\Drupal::translation()
          ->formatPlural($result['@numitems'], $singular_message, $plural_message, $result));
      }
    }
  }

  /**
   * {@inheritdoc}
   */
  public function checkStatus() {
    $status = parent::checkStatus();
    if ($status == MigrationInterface::RESULT_COMPLETED) {

      // Do some batch housekeeping.
      $context = $this
        ->getBatchContext();
      if (!empty($context['sandbox']) && $context['sandbox']['operation'] == MigrateBatchExecutable::BATCH_IMPORT) {
        $context['sandbox']['batch_counter']++;
        if ($context['sandbox']['batch_counter'] >= $context['sandbox']['batch_limit']) {
          $status = MigrationInterface::RESULT_INCOMPLETE;
        }
      }
    }
    return $status;
  }

  /**
   * Calculates how much a single batch iteration will handle.
   *
   * @param array|\DrushBatchContext $context
   *   The sandbox context.
   *
   * @return float
   *   The batch limit.
   */
  public function calculateBatchLimit($context) {

    // TODO Maybe we need some other more sophisticated logic here?
    return ceil($context['sandbox']['total'] / 100);
  }

}

Members

Namesort descending Modifiers Type Description Overrides
MigrateBatchExecutable::$batchContext protected property The current batch context.
MigrateBatchExecutable::$checkDependencies protected property Indicates if we need import dependent migrations also.
MigrateBatchExecutable::$migrationPluginManager protected property Plugin manager for migration plugins.
MigrateBatchExecutable::$updateExistingRows protected property Indicates if we need to update existing rows or skip them.
MigrateBatchExecutable::batchFinishedImport public static function Finished callback for import batches.
MigrateBatchExecutable::batchImport public function Setup batch operations for running the migration.
MigrateBatchExecutable::batchOperations protected function Helper to generate the batch operations for importing migrations.
MigrateBatchExecutable::batchProcessImport public static function Batch 'operation' callback.
MigrateBatchExecutable::BATCH_IMPORT constant Representing a batch import operation.
MigrateBatchExecutable::calculateBatchLimit public function Calculates how much a single batch iteration will handle.
MigrateBatchExecutable::checkStatus public function Checks for exceptional conditions, and display feedback. Overrides MigrateExecutable::checkStatus
MigrateBatchExecutable::getBatchContext public function Gets a reference to the current batch context.
MigrateBatchExecutable::setBatchContext public function Sets the current batch content so listeners can update the messages.
MigrateBatchExecutable::__construct public function Constructs a MigrateExecutable and verifies and sets the memory limit. Overrides MigrateExecutable::__construct
MigrateExecutable::$counter protected property Count of number of items processed so far in this migration.
MigrateExecutable::$counts protected property An array of counts. Initially used for cache hit/miss tracking.
MigrateExecutable::$deleteCounter protected property Counter of map deletions.
MigrateExecutable::$eventDispatcher protected property The event dispatcher.
MigrateExecutable::$feedback protected property Frequency (in items) at which progress messages should be emitted.
MigrateExecutable::$idlist protected property List of specific source IDs to import.
MigrateExecutable::$itemLimit protected property Maximum number of items to process in this migration.
MigrateExecutable::$itemLimitCounter protected property Counter of map saves, used to detect the item limit threshold.
MigrateExecutable::$listeners protected property List of event listeners we have registered.
MigrateExecutable::$memoryLimit protected property The PHP memory_limit expressed in bytes.
MigrateExecutable::$memoryThreshold protected property The ratio of the memory limit at which an operation will be interrupted.
MigrateExecutable::$message public property Migration message service.
MigrateExecutable::$migration protected property The configuration of the migration to do.
MigrateExecutable::$preExistingItem protected property Whether the destination item exists before saving.
MigrateExecutable::$saveCounters protected property Counters of map statuses.
MigrateExecutable::$source protected property The source.
MigrateExecutable::$sourceIdValues protected property The configuration values of the source.
MigrateExecutable::$sourceRowStatus protected property Status of one row.
MigrateExecutable::attemptMemoryReclaim protected function Tries to reclaim memory. 1
MigrateExecutable::currentSourceIds protected function Fetches the key array for the current source record.
MigrateExecutable::formatSize protected function Generates a string representation for the given byte count. 1
MigrateExecutable::getCreatedCount public function Return the number of items created.
MigrateExecutable::getEventDispatcher protected function Gets the event dispatcher.
MigrateExecutable::getFailedCount public function Return the number of items that failed.
MigrateExecutable::getIdMap protected function Get the ID map from the current migration. Overrides MigrateExecutable::getIdMap
MigrateExecutable::getIgnoredCount public function Return the number of items ignored.
MigrateExecutable::getMemoryUsage protected function Returns the memory usage so far. 1
MigrateExecutable::getProcessedCount public function Return the total number of items processed.
MigrateExecutable::getRollbackCount public function Return the number of items rolled back.
MigrateExecutable::getSource protected function Returns the source. Overrides MigrateExecutable::getSource
MigrateExecutable::getUpdatedCount public function Return the number of items updated.
MigrateExecutable::handleException protected function Takes an Exception object and both saves and displays it. 1
MigrateExecutable::import public function Performs an import operation - migrate items from source to destination. Overrides MigrateExecutableInterface::import
MigrateExecutable::memoryExceeded protected function Tests whether we've exceeded the desired memory threshold. 1
MigrateExecutable::onMapDelete public function Count up any rollback events.
MigrateExecutable::onMapSave public function Count up any map save events.
MigrateExecutable::onPostImport public function React to migration completion.
MigrateExecutable::onPostRollback public function React to rollback completion.
MigrateExecutable::onPostRowDelete public function React to item rollback.
MigrateExecutable::onPrepareRow public function React to a new row.
MigrateExecutable::onPreRowSave public function React to an item about to be imported.
MigrateExecutable::processRow public function Processes a row. Overrides MigrateExecutableInterface::processRow
MigrateExecutable::progressMessage protected function Emit information on what we've done.
MigrateExecutable::removeListeners protected function Clean up all our event listeners.
MigrateExecutable::resetCounters protected function Reset all the per-status counters to 0.
MigrateExecutable::rollback public function Performs a rollback operation - remove previously-imported items. Overrides MigrateExecutableInterface::rollback
MigrateExecutable::rollbackMessage protected function Emit information on what we've done.
MigrateExecutable::saveMessage public function Passes messages through to the map class. Overrides MigrateExecutableInterface::saveMessage
StringTranslationTrait::$stringTranslation protected property The string translation service. 1
StringTranslationTrait::formatPlural protected function Formats a string containing a count of items.
StringTranslationTrait::getNumberOfPlurals protected function Returns the number of plurals supported by a given language.
StringTranslationTrait::getStringTranslation protected function Gets the string translation service.
StringTranslationTrait::setStringTranslation public function Sets the string translation service to use. 2
StringTranslationTrait::t protected function Translates a string to the current language or to a given language.