Importing/Exporting content between Dev & Prod

[patrickkaleta]

NOTE: This issue discusses how to migrate actual content (nodes like data packages, resources etc. and taxonomy terms like the EU-GL taxonomy for example). Synchronizing configuration is discussed in issue #138 and needs to be completed BEFORE synchronizing content.

Overview

For synchronizing content we use the Migrate module, which is part of the Drupal Core and some of its additional modules (Migrate Plus and Migrate Tools). The content is pulled from the Dev via REST-Views (all called "Drupal-shared ...", which means that Dev needs to be accessible via HTTP.

The general workflow will be that certain nodes (GL-step templates, data packages and all their referenced entities) and all of the taxonomies will be exported in Dev and imported in Prod. The following actions are performed:

• New elements will be created in Prod
• Existing elements will be updated
• Elements removed in Dev will also be removed in Prod
• For synchronizing only published elements will be taken into account

Managing the migration configuration files

The target site (in our case the production system) needs for each content type the migration configuration stored in a .yml file (these .yml files are stored and synchronized with the other configuration .yml files, see #138). To add a new migration configuration:

1. Create the yml file (use existing ones for reference)
2. Add the file in the Drupal backend UI at .../admin/config/development/configuration/single/import (as configuration type select "Migration")
3. Synchronize configuration between Dev and Prod (Importing/Exporting configuration between Dev and Prod environment #138)
4. If an existing migration needs to be changed, use the "Single item export" feature under same link as in (2) and export desired migration including the migration UUID, which will be needed for the re-import later after changes to file are done

Synchronization process

1. in the Docker-container of Prod run either the provided bash script to synchronize all content in the correct order
2. or synchronize individual content types using drush (make sure to migrate referenced items first!):
   drush mim [ID_of_Migration_file] --update

Things to know

• --update argument can be omitted. In that case existing items will not be updated but instead be skipped
• in case a migration gets stuck, its status can be reset with drush mrs [ID_of_Migration_file]
• should a migration fail (e.g. content was removed in Prod and can't be found for lookup process) the migration can be rolled back with drush mr [ID_of_Migration_file]
• --all argument can be used with drush mr or drush mrs to take affect on all migrations and not just one
• drush ms lists all available migrations and their status and other details (can also be seen in Drupal BE)
• under Structure -> Migrations (.../admin/structure/migrate) all migrations can also be viewed in the BE. It's also possible to remove individual migrations there.

[p-a-s-c-a-l]

@therter Can you please additionally explain the role of republish.sh in this process? E.g. when is it run, on DEV, on PROD, on both, ...?

[therter]

@therter Can you please additionally explain the role of republish.sh in this process? E.g. when is it run, on DEV, on PROD, on both, ...?

This script should be executed before the synchronisation process starts. So far, it was started manually by @patrickkaleta to publish the available cities/countries.
I am not sure, if it is enough to execute the script on the dev instance and the cities will be updated on the prod instance during the synchronisation process or if the script should also be executed on the prod instance.