Over the past few months working on migrations to Drupal 8, researching best practices, and contributing to core and contributed modules, I discovered that there are several tools available in core and contributed modules, plus a myriad of how-to articles. To save you the trouble of pulling it all together yourself, I offer a comprehensive overview of the Migrate module plus a few other contributed modules that complement it in order to migrate a Drupal site to 8.
Let's begin with the most basic element: migration files.
Migration files
In Drupal 8, the migrate module splits a migration into a set of YAML files, each of them is composed by a source (like the node table in a Drupal 7 database), a process (the field mapping and processing), and a destination (like a node entity in 8). Here is a subset of the migration files present in core:
$ find core -name *.yml | grep migrations
core/modules/statistics/migrations/statistics_settings.yml
core/modules/statistics/migrations/statistics_node_counter.yml
core/modules/shortcut/migrations/d7_shortcut_set.yml
core/modules/shortcut/migrations/d7_shortcut.yml
core/modules/shortcut/migrations/d7_shortcut_set_users.yml
core/modules/tracker/migrations/d7_tracker_node.yml
core/modules/tracker/migrations/d7_tracker_settings.yml
core/modules/tracker/migrations/d7_tracker_user.yml
core/modules/path/migrations/d7_url_alias.yml
...
And below is the node migration, located at core/modules/node/migrations/d7_node.yml:
id: d7_node
label: Nodes
audit: true
migration_tags:
- Drupal 7
- Content
deriver: Drupal\node\Plugin\migrate\D7NodeDeriver
source:
plugin: d7_node
process:
nid: tnid
vid: vid
langcode:
plugin: default_value
source: language
default_value: "und"
title: title
uid: node_uid
status: status
created: created
changed: changed
promote: promote
sticky: sticky
revision_uid: revision_uid
revision_log: log
revision_timestamp: timestamp
destination:
plugin: entity:node
migration_dependencies:
required:
- d7_user
- d7_node_type
optional:
- d7_field_instance
- d7_comment_field_instance
Migration files can be generated dynamically via a deriver like the node migration defines above, which uses D7NodeDeriver to generate a migration for each content type’s data and revision tables. On top of that, migrations can be classified via the migration_tags section (the migration above has the Drupal 7 and Content tags).
Configuration vs content migrations
Migration files may have one or more tags to classify them. These tags are used for running them in groups. Some migration files may have the Configuration tag—like the node type or field migrations—while others might have the Content tag—like the node or user migrations. Usually, you would run the Configuration migrations first in order to configure the new site, and then the Content ones so the content gets fetched, transformed, and inserted on top of such configuration.
Notice though that depending on how much you are planning to change the content model, you may decide to configure the new site manually and write the Content migrations by hand. In the next section, we will examine the differences between generating and writing migrations.
Generating vs Writing migrations
Migration files living within the core, contributed, and custom modules are static files that need to be read and imported into the database as migration plugins so they can be executed. This process, depending on the project needs, can be implemented in two different ways:
Using Migrate Upgrade
Migrate Upgrade module implements a Drush command to automatically generate migrations for all the configuration and content in an existing Drupal 6 or 7 site. The best thing about this approach is that you don’t have to manually create the new content model in the new site since Migrate Upgrade will inspect the source database and do it for you by generating the migrations.
If the existing content model won’t need to go through major changes during the migration, then Migrate Upgrade is a great choice to generate migrations. There is an API that developers can interact with in order to alter migrations and the data being processed. We will see a few examples further down in this article.
Writing migrations by hand
If the content model will go through a deep reorganization such as merging content from different sources into one, reorganizing fields, and changing machine names, then configuring the new site manually and writing content migrations may be the best option. In this scenario, you would write the migration files directly to the config/sync directory so then they can be imported via drush config:import and executed via drush migrate:import.
Notice that if the content model has many entity types, bundles, and fields, this can be a tough job so even if the team decides to go this route, generating content migrations with Migrate Upgrade can be useful since the resulting migrations can serve as templates for the ones to be written.
Setting up the new site for running migrations
Assuming that we have a new site created using the Composer Drupal Project and we have run the installer, we need to require and install the following modules:
$ composer require drupal/migrate_tools drupal/migrate_upgrade drupal/migrate_plus
$ drush pm:enable --yes migrate_tools,migrate_upgrade,migrate_plus
Next, we need to add a database connection to the old site, which we would do at web/sites/default/settings.local.php:
// The default database connection details.
$databases['default']['default'] = [
'database' => 'drupal8',
'username' => 'root',
'password' => 'root',
'prefix' => '',
'host' => '127.0.0.1',
'port' => '3306',
'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
'driver' => 'mysql',
];
// The Drupal 7 database connection details.
$databases['drupal7']['default'] = [
'database' => 'drupal7',
'username' => 'root',
'password' => 'root',
'prefix' => '',
'host' => '127.0.0.1',
'port' => '3306',
'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
'driver' => 'mysql',
];
With the above setup, we can move on to the next step, where we will generate migrations.
Generating migrations with Migrate Upgrade
The following command will read all migration files, create migrate entities out of them and insert them into the database so they are ready to be executed:
$ drush --yes migrate:upgrade --legacy-db-key drupal7 --legacy-root sites/default/files --configure-only
It is a good practice to export the resulting migrate entities as configuration so we can track their changes. Therefore, we will export configuration after running the above command, which will create a list of files like the following:
$ drush --yes config:export
[notice] Differences of the active config to the export directory:
+------------------------------------------------------------------+-----------+
| Config | Operation |
+------------------------------------------------------------------+-----------+
| migrate_plus.migration.upgrade_d7_date_formats | Create |
| migrate_plus.migration.upgrade_d7_field | Create |
| migrate_plus.migration.upgrade_d7_field_formatter_settings | Create |
| migrate_plus.migration.upgrade_d7_field_instance | Create |
| migrate_plus.migration.upgrade_d7_field_instance_widget_settings | Create |
| migrate_plus.migration.upgrade_d7_file | Create |
| migrate_plus.migration.upgrade_d7_filter_format | Create |
| migrate_plus.migration.upgrade_d7_filter_settings | Create |
| migrate_plus.migration.upgrade_d7_image_styles | Create |
| migrate_plus.migration.upgrade_d7_menu | Create |
| migrate_plus.migration.upgrade_d7_menu_links | Create |
| migrate_plus.migration.upgrade_d7_node_article | Create |
| migrate_plus.migration.upgrade_d7_node_revision_article | Create |
...
In the above list, we see a mix of Configuration and Content migrations being created. Now we can check the status of each migration via the migrate:status Drush command:
$ drush migrate:status
-----------------------------------------------------------------------------------------------------
Migration ID Status Total Imported Unprocessed Last Imported
-----------------------------------------------------------------------------------------------------
upgrade_d7_date_formats Idle 7 0 0
upgrade_d7_filter_settings Idle 1 0 0
upgrade_d7_image_styles Idle 193 0 0
upgrade_d7_node_settings Idle 1 0 0
upgrade_d7_system_date Idle 1 0 0
upgrade_d7_url_alias Idle 25981 0 0
upgrade_system_site Idle 1 0 0
upgrade_taxonomy_settings Idle 0 0 0
upgrade_d7_path_redirect Idle 518 0 0
upgrade_d7_field Idle 253 0 0
upgrade_d7_field_collection_type Idle 9 0 0
upgrade_d7_node_type Idle 16 0 0
upgrade_d7_taxonomy_vocabulary Idle 34 0 0
upgrade_d7_field_instance Idle 738 0 0
upgrade_d7_view_modes Idle 24 0 0
upgrade_d7_field_formatter_settings Idle 1280 0 0
upgrade_d7_field_instance_widget_settings Idle 738 0 0
upgrade_d7_file Idle 5731 0 0
upgrade_d7_filter_format Idle 6 0 0
upgrade_d7_menu Idle 5 0 0
upgrade_d7_user_role Idle 6 0 0
upgrade_d7_user Idle 82 0 0
upgrade_d7_node_article Idle 322400 0 0
upgrade_d7_node_page Idle 342 0 0
upgrade_d7_menu_links Idle 623 0 0
upgrade_d7_node_revision_article Idle 742577 0 0
upgrade_d7_node_revision_page Idle 2122 0 0
upgrade_d7_taxonomy_term_tags Idle 1729 0 0
-------------------------------------------- -------- ------- ---------- ------------- ---------------
You should inspect any contributed modules in use on the old site and install them in the new one as they may contain migrations. For example, if the old site is using Redirect module, by installing itm and generating migrations like we did above, you should see a new migration provided by this module ready to go.
Running migrations
Assuming that we decided to run migrations generated by Migrate Upgrade (otherwise skip the following sub-section), we would first run configuration migrations and then content ones.
Configuration migrations
Here is the command to run all migrations with the tag Configuration along with their dependencies.
$ drush migrate:import --tag=Configuration --execute-dependencies
[notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_date_formats'
[notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_settings'
[notice] Processed 20 items (20 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_styles'
[notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_settings'
[notice] Processed 1 items (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_date'
[notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_site'
[notice] Processed 100 items (100 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field'
[notice] Processed 200 items (200 created, 0 updated, 0 failed, 379 ignored) - done with 'upgrade_d7_field_instance'
...
The above command will migrate site configuration, content types, taxonomy vocabularies, view modes, and such. Once this is done, it is recommended to export the resulting configuration via drush config:export and commit the changes. From then on, if we make changes in the old site’s configuration or we alter the Configuration migrations (we will see how further down), we will need to roll back the affected migrations and run them again.
For example, the Media Migration module creates media fields and alters field mappings in content migrations so after installing it we should run the following commands to re-run them:
$ drush --yes migrate:rollback upgrade_d7_view_modes,upgrade_d7_field_instance_widget_settings,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance,upgrade_d7_field
$ drush --yes migrate:import --execute-dependencies upgrade_d7_field,upgrade_d7_field_instance,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance_widget_settings,upgrade_d7_view_modes
$ drush --yes config:export
Once we have executed all Configuration migrations, we can run content ones.
Content migrations
Content migrations are straightforward. We can run them with the following command:
$ drush migrate:import --tag=Content --execute-dependencies
Logging and tracking
Migrate keeps track of all the migrated content via the migrate_* tables. If you check out the new database after running migrations, you will see something like this:
- A set of migrate_map* tables, storing the old and new identifiers of each migrated entity. These tables are used by the
migrate:rollbackDrush command to roll back migrated data. - A set of migrate_messages* tables, which hold errors and warnings that occurred while running migrations. These can be seen via the
migrate:messagesDrush command.
Rolling back migrations
In the previous section, we rolled back the field migrations before running them again. This process is great for reverting imported Configuration or Content, which you will do often while developing a migration.
Here is an example. Let’s suppose that you have executed content migrations but articles did not get migrated as you expected. The process you would follow to fix them would be:
- Find the migration name via
drush migrate:status | grep article. - Roll back migrations with
drush migrate:rollback upgrade_d7_node_revision_article,upgrade_d7_node_article. - Perform the changes that you need either directly at the exported migration at
config/syncor by altering them and then recreating them withmigrate:upgradelike we did at Generating migrations with Migrate Upgrade. We will see how to alter migrations in the next section. - Run the migrations again with
drush migrate:import upgrade_d7_node_article,upgrade_d7_node_revision_article. - Verify the changes and, if needed, repeat steps 2 to 4 until you are done.
Migrate events and hooks
Before diving into the APIs to alter migrations, let’s clarify that there are two processes that we can hook into:
- The
migrate:upgradeDrush command, which reads all migration files in core, contributed, and custom modules and imports them into the database. - The
migrate:importDrush command, which runs migrations.
In the next sub-sections, we will see how can we interact with these two commands.
Altering migrations (migrate:upgrade)
Drupal core offers hook_migration_plugins_alter(), which receives the array of available migrations that migrate:upgrade creates. Here is a sample implementation at mymodule.module where we delegate the logic to a service:
/**
* Implements hook_migration_plugins_alter().
*/
function mymodule_migration_plugins_alter(array &$migrations) {
$migration_alterer = \Drupal::service('mymodule.migrate.alterer');
$migration_alterer->process($migrations);
}
And here is a subset of the contents of the service:
class MigrationAlterer {
/**
* Processes migration plugins.
*
* @param array $migrations
* The array of migration plugins.
*/
public function process(array &$migrations) {
$this->skipMigrations($migrations);
$this->disablePathautoAliasCreation($migrations);
$this->setContentLangcode($migrations);
$this->setModerationState($migrations);
$this->persistUuid($migrations);
$this->skipFileCopy($migrations);
$this->alterRedirect($migrations);
}
/**
* Skips unneeded migrations.
*
* @param array $migrations
* The array of migration plugins.
*/
private function skipMigrations(array &$migrations) {
// Skip unwanted migrations.
$migrations_to_skip = [
'd7_block',
'd7_comment',
'd7_comment_entity_display',
'd7_comment_entity_form_display_subject',
'd7_comment_field',
'd7_comment_entity_form_display',
'd7_comment_type',
'd7_comment_field_instance',
'd7_contact_settings',
];
$migrations = array_filter($migrations, function ($migration) use ($migrations_to_skip) {
return !in_array($migration['id'], $migrations_to_skip);
});
}
// The remaining methods would go here.
}
In the next section, we will see how to alter the data being migrated while running migrations.
Altering data while running migrations (migrate:import)
Drupal core offers hook_migrate_prepare_row() and hook_migrate_MIGRATION_ID_prepare_row, which are triggered before each row of data is processed by the migrate:import Drush command. Additionally, there is a set of events that we can subscribe to such as before and after the migration starts or before and after a row is saved.
On top of the above, Migrate Plus module exposes an event that wraps hook_migrate_prepare_row(). Here is a sample subscriber for this event:
class MyModuleMigrationSubscriber implements EventSubscriberInterface {
/**
* Prepare row event handler.
*
* @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
* The migrate row event.
*
* @throws \Drupal\migrate\MigrateSkipRowException
* If the row needs to be skipped.
*/
public function onPrepareRow(MigratePrepareRowEvent $event) {
$this->alterFieldMigrations($event);
$this->skipMenuLinks($event);
$this->setContentModeration($event);
}
/**
* Alters field migrations.
*
* @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
* The migrate row event.
*
* @throws \Drupal\migrate\MigrateSkipRowException
* If a row needs to be skipped.
* @throws \Exception
* If the source cannot be changed.
*/
private function alterFieldMigrations(MigratePrepareRowEvent $event) {
$field_migrations = [
'upgrade_d7_field',
'upgrade_d7_field_instance',
'upgrade_d7_view_modes',
'upgrade_d7_field_formatter_settings',
'upgrade_d7_field_instance_widget_settings',
];
if (in_array($event->getMigration()->getPluginId(), $field_migrations)) {
// Here are calls to private methods that alter these migrations.
}
}
/**
* Skips menu links that are either implemented or not needed.
*
* @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
* The migrate row event.
*
* @throws \Drupal\migrate\MigrateSkipRowException
* If a row needs to be skipped.
*/
private function skipMenuLinks(MigratePrepareRowEvent $event) {
if ($event->getMigration()->getPluginId() != 'upgrade_d7_menu_links') {
return;
}
$paths_to_skip = [
'some/path',
'other/path',
];
$menu_link = $event->getRow()->getSourceProperty('link_path');
if (in_array($menu_link, $paths_to_skip)) {
throw new MigrateSkipRowException('Skipping menu link ' . $menu_link);
}
}
/**
* Sets the content moderation field on node migrations.
*
* @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
* The migrate row event.
*
* @throws \Exception
* If the source cannot be changed.
*/
private function setContentModeration(MigratePrepareRowEvent $event) {
$row = $event->getRow();
$source = $event->getSource();
if (('d7_node' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) {
$state = $row->getSourceProperty('status') ? 'published' : 'draft';
$row->setSourceProperty('moderation_state', $state);
}
elseif (('d7_node_revision' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) {
$row->setSourceProperty('moderation_state', 'draft');
}
}
}
Conclusion
When you are migrating a Drupal site to 8, Migrate Upgrade module does a lot of the work for free by generating Configuration and Content migrations. Even if you decide to write any of them by hand, it is convenient to run the command and use the resulting migrations as a template for your own. In the next article, we will see how to work seamlessly on a migration while the rest of the team is building the new site.
Thanks to Andrew Berry, April Sides, Karen Stevenson, Marcos Cano, and Salvador Moreno for their feedback and help. Photo by Barth Bailey on Unsplash.