Updating a D6 -> Drupal 8.0.x migration to Drupal 8.1.x

We have several Drupal 6 to Drupal 8 upgrade projects going on, which is particularly challenging given how quickly the Drupal Migration system is changing. Given that a couple of them are nearing launch, and were missing some node references, I set out to get the content updated from the production sites before launch. When we started migrating these sites, you could run the Migrate Upgrade scripts passing the database credentials, and get a full list of what could be upgraded and what could not. And it would migrate all the stuff. The whole point of having a migration system like this is so that you can keep on incrementally migrating content until the new site is ready to go, do a final migration, and go live! The first surprise: with migration in 8.1.x, that is no longer supported with the built-in automatic Drupal to Drupal upgrade -- it's a one-shot, one way deal. Oh crap. Fortunately, it's pretty easy to use migrate_upgrade to generate a custom migration, and with some database mangling, you can get your new migration to pick up the mappings from a previous auto-migration done in 8.0.x. The main gotcha here is that node content still isn't showing up as migrated -- but for us, that was good enough -- that's what we wanted to update from production anyway. The big win here is that all of the rest of the dependencies for the migration -- users, input filters, taxonomy terms, blocks, menus, settings -- are all now mapped and so we can keep the changes we've made on the new site without clobbering with an entirely new migration.

Caution! Extremely technical backflips ahead...

Ok, not really... but this is tricky stuff, involving multiple wipes/reloading of your database. Make sure you do this all on a test copy!

Before starting: Gather your tools, backups

Before doing anything, I would recommend backing up your database. And make sure your tools are up to date. Here are the tools I use for this process:

• drush 8.x
• drupal console 1.0.0 (currently at beta3)
• bash, sed
• a working Drupal 8.0 site (it's ok if you've already upgraded it to 8.1)

Step 1: Back up database, upgrade all code

drush sql-dump > site-80.sql
(do whatever you usually do to upgrade)
drush dl --select migrate_tools migrate_upgrade migrate_plus # select the latest dev versions of these
drush updb
drush cr # cache rebuild
drush sql-dump > site-81.sql
```

Note: for some reason, drush dl migrate_upgrade wants to install in ~/.drush, and not in the site -- this caused me a bit of confusion, and lots of errors when trying to update! Be sure it installs in a location that doesn't get overridden by the 1.x version, which is not compatible with Drupal 8.1.

For changes as major as these, I like to have places I can roll back to. git tags can be useful, along with database snapshots at each step.

### Step 2: Save your config

The config by default is stored under sites/default/files -- we exclude that path from git, and so we recommend moving it somewhere inside your git tree. You can set this in the settings.php file, to whatever path you would like:

```
$config_directories['sync'] = 'config/sync';
```

... the above configuration puts it under the site root, in config/sync.

Once you have that set:

```
drupal config:export
git add config
git commit
```

... and you have your current configuration exported and saved to code.

### Step 3: Apply any core patches, prep the destination

In particular, we wanted node references migrated to D8 entity references: [Add migrate support for Drupal 6 node & user reference](https://www.drupal.org/node/2447727)

At this point you should enable all the modules that have migrations you want picked up, and generally make sure your target environment is ready.

If you make any further changes, export and commit your config again, and create a new database backup BEFORE creating your new upgrade.

### Step 4: Create your custom migration module, and create the new migration

Here I borrowed heavily from other posts:

* https://drupalize.me/blog/201606/custom-drupal-drupal-migrations-migrate-tools
* https://www.drupaleasy.com/blogs/ultimike/2016/04/drupal-6-drupal-81x-custom-content-migration

The gist of it:

```
drupal generate:module # Create a new module to contain the migration configuration
gedit sites/default/settings # Add another database credential array for the D6 database, using a key "upgrade"
drush migrate-upgrade --configure-only # Creates all the migration config objects it can find
drupal config:export
```

At this point you have a brand new migration in the config, and a module ready to store this config. Assuming your module is in module/custom/site_migrate:

```
mkdir -p module/custom/site_migrate/config/install
cp config/sync/migrate_plus.migration* module/custom/site_migrate/config/install/
rm module/custom/site_migrate/config/install/migrate_plus.migration_group.default.yml
```

... this exports your migration config to code, skipping the default migration_group that would conflict with migrate_plus.

### Step 5: Mangle all the data

Ok. Now we're to the tricky part. We're essentially going to revert to where we were at the end of step 3, and then enable our migration module, with the mapping tables from the original migration.

So...

```
git add modules/custom/site_module
git commit
git clean -df config/sync
git checkout HEAD config/sync
```

... to get your code base back to the right spot.

Now, the guts of the change. The new migrate_upgrade uses mostly the same migration names as the old, except with upgrade_ inserted at the beginning of the name. This means you need to rename your migrate_map and migrate_message tables to pick up those original mappings.

There's one other gotcha: at the moment, drush migrate-upgrade looks for a database key named "upgrade" but in the exported configuration, it changes to expect a database key named "drupal_6". So you'll need to edit your setting.php file to change that, after step 4.

Here's how I renamed the tables. Exactly what you need to change may vary based on the version of your mysql libraries and drush, but this worked for me:

```
cp site-81.sql site-mm.sql
sed -i "s/ALTER TABLE `migrate_message_/ALTER TABLE `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/ALTER TABLE `migrate_map_/ALTER TABLE `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^LOCK TABLES `migrate_message_/LOCK TABLES `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^LOCK TABLES `migrate_map_/LOCK TABLES `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^INSERT INTO `migrate_message_/INSERT INTO `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^INSERT INTO `migrate_map_/INSERT INTO `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^CREATE TABLE `migrate_message_/CREATE TABLE `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^CREATE TABLE `migrate_map_/CREATE TABLE `migrate_map_upgrade_/g" site-mm.sql 
```

### Step 6: Run your migration!

Almost there...

```
drupal database:drop # this deletes any extraneous tables that might cause issues
drush sqlc