Migrate API in Drupal 9

Same name and namespace in other branches

1. 8 core/modules/migrate/migrate.api.php \migration

Overview of the Migrate API, which migrates data into Drupal.

Overview of a migration

Migration is an Extract, Transform, Load (ETL) process. In the Drupal Migrate API, the extract phase is called 'source', the transform phase is called 'process', and the load phase is called 'destination'. It is important to understand that the term 'load' in ETL refers to loading data into the storage while in a typical Drupal context the term 'load' refers to loading data from storage.

In the source phase, a set of data, called the row, is retrieved from the data source. The data can be migrated from a database, loaded from a file (for example CSV, JSON or XML) or fetched from a web service (for example RSS or REST). The row is sent to the process phase where it is transformed as needed or marked to be skipped. After processing, the transformed row is passed to the destination phase where it is loaded (saved) into the target Drupal site.

Migrate API uses the Drupal plugin system for many different purposes. Most importantly, the overall ETL process is defined as a migration plugin and the three phases (source, process and destination) have their own plugin types.

Migrate API migration plugins

Migration plugin definitions are stored in a module's 'migrations' directory. The plugin class is \Drupal\migrate\Plugin\Migration, with interface \Drupal\migrate\Plugin\MigrationInterface. Migration plugins are managed by the \Drupal\migrate\Plugin\MigrationPluginManager class. Migration plugins are only available if the providers of their source plugins are installed.

Example migrations in Migrate API handbook.

Migrate API source plugins

Migrate API source plugins implement \Drupal\migrate\Plugin\MigrateSourceInterface and usually extend \Drupal\migrate\Plugin\migrate\source\SourcePluginBase. They are annotated with \Drupal\migrate\Annotation\MigrateSource annotation and must be in namespace subdirectory 'Plugin\migrate\source' under the namespace of the module that defines them. Migrate API source plugins are managed by the \Drupal\migrate\Plugin\MigrateSourcePluginManager class.

List of source plugins provided by the core Migrate module. Core and contributed source plugin usage examples in Migrate API handbook.

Migrate API process plugins

Migrate API process plugins implement \Drupal\migrate\Plugin\MigrateProcessInterface and usually extend \Drupal\migrate\ProcessPluginBase. They are annotated with \Drupal\migrate\Annotation\MigrateProcessPlugin annotation and must be in namespace subdirectory 'Plugin\migrate\process' under the namespace of the module that defines them. Migrate API process plugins are managed by the \Drupal\migrate\Plugin\MigratePluginManager class.

List of process plugins for common operations provided by the core Migrate module.

Migrate API destination plugins

Migrate API destination plugins implement \Drupal\migrate\Plugin\MigrateDestinationInterface and usually extend \Drupal\migrate\Plugin\migrate\destination\DestinationBase. They are annotated with \Drupal\migrate\Annotation\MigrateDestination annotation and must be in namespace subdirectory 'Plugin\migrate\destination' under the namespace of the module that defines them. Migrate API destination plugins are managed by the \Drupal\migrate\Plugin\MigrateDestinationPluginManager class.

List of destination plugins for Drupal configuration and content entities provided by the core Migrate module.

Migrate API key concepts

Stubs

Taxonomy terms are an example of a data structure where an entity can have a reference to a parent. When a term is being migrated, it is possible that its parent term has not yet been migrated. Migrate API addresses this 'chicken and egg' dilemma by creating a stub term for the parent so that the child term can establish a reference to it. When the parent term is eventually migrated, Migrate API updates the previously created stub with the actual content.

Map tables

Once a migrated row is saved and the destination IDs are known, Migrate API saves the source IDs, destination IDs, and the row hash into a map table. The source IDs and the hash facilitate tracking changes for continuous migrations. Other migrations can use the map tables for lookup purposes when establishing relationships between records.

Highwater mark

A Highwater mark allows the Migrate API to track changes so that only data that has been created or updated in the source since the migration was previously executed is migrated. The only requirement to use the highwater feature is to declare the row property to use for the highwater mark. This can be any property that indicates the highest value migrated so far. For example, a timestamp property that indicates when a row of data was created or last updated would make an excellent highwater property. If the migration is executed again, only those rows that have a higher timestamp than in the previous migration would be included.

source:
  plugin: d7_node
  high_water_property:
    name: changed

In this example, the row property 'changed' is the high_water_property. If the value of 'changed' is greater than the current highwater mark the row is processed and the value of the highwater mark is updated to the value of 'changed'.

Rollbacks

When developing a migration, it is quite typical that the first version does not provide correct results for all migrated data. Rollbacks allow you to undo a migration and then execute it again after adjusting it.

Documentation handbooks

Migrate API handbook. Upgrading to Drupal 8 handbook.

File

core/modules/migrate/migrate.api.php, line 12

Hooks provided by the Migrate module.

Functions

Classes

Interfaces