What is a site deployment module?

In a Drupal development-staging-production workflow, the best practice is for new features and bug fixes to be developed locally, then moved downstream to the staging environment, and later to production.

Just how changes are pushed downstream varies, but typically the process includes Features, manual changes to the production user interface, drush commands, and written procedures.

Some examples include:

  • A view which is part of a Feature called xyz_feature is modified; the feature is updated and pushed to the git repo; and then the feature is reverted using drush fr xyz_feature on the production site.
  • A new default theme is added to the development site and tested, and pushed to the git repo; and then the new theme is selected as default on the production site's admin/appearance page.
  • Javascript aggregation is set on the dev site's admin/config/development/performance page, and once everything works locally, it is set on the production via the user interface.

This approach is characterized by the following properties:

  • Each incremental deployment is different and must be documented as such.
  • If there exist several environments, one must keep track manually of what "remains to be done" on each environment.
  • The production database is regularly cloned downstream to a staging environment, but it is impossible to tell when was the last time it was cloned.
  • If an environment is out of date and does not contain any important data, it can be deleted and the staging environment can be re-cloned.
  • Many features (for example javascript aggregation) are never in version control, at best only documented in an out-of-date wiki, at worst in the memory of a long-gone developer.
  • New developers clone the staging database to create a local development environment.
  • Automated functional testing by a continuous integration server, if done at all, uses a clone of the staging database.

The main issue I have with this approach is that it overly relies on the database to store important configuration, and the database is not under version control. There is no way to tell who did what, and when.

The deployment module

Using a deployment module aims to meet the following goals:

  • Everything except content should be in version control: views, the default theme, settings like Javascript aggregation, etc.
  • Incremental deployments should always be performed following the same procedure.
  • Initial deployments (for example for a new developer or for a throwaway environment during an automated test) should be possible without cloning the database.
  • Tests should be run agains a known-good starting point, not a clone of a database.
  • New developers should be up and running without having to clone a database.

Essentially, anything not in version control is unreliable, and cloning the database today can yield a bug which won't be present if you clone the database tomorrow. So we'll avoid cloning the database in most cases.

The Dcycle manifesto states that each site should have a deployment module whose job it is to keep track of deployment-related configuration. Once you have settled on a namespace for your project, for example example, by convention your deployment module should reside in sites/*/modules/custom/example_deploy.

Let's now say that we are starting a project, and our first order of business is to create a specific view: we will create the view, export it as a feature, and make the feature a dependency of our deployment module. Starting now, if all your code is under version control, all new environments (production, continuous integration, testing, new local sites) are deployed the same way, simply by creating a database and enabling the deployment module. Using Drush, you would call something like:

echo 'create database example' | mysql -uroot -proot
drush si --db-url=mysql://root:root@localhost/example --account-name=root --account-pass=root
drush en example_deploy -y

The first line creates the database; the second line is the equivalent of clicking though Drupal's installation procedure; and the third line activates the deployment module.

Because you have set your feature to be a dependency of your deployment module, it is activated and your view is deployed.

Incremental deployments

We want the incremental deployment procedure to always be the same. Also, we don't want it to involve cloning the database, because the database is in an unknown state (it is not under version control). Another reason we don't want to clone the database is because we want to practice our incremental deployment procedure as must as possible, ideally several times a day, to catch any problems before we apply it to the production site.

My incremental deployment procedure, for all my Drupal projects, uses Drush and Registry rebuild, and goes as follows once the new code has been fetched via git:

drush rr
drush vset maintenance_mode 1
drush updb -y
drush cc all
drush cron
drush vset maintenance_mode 0

The first line (drush rr) rebuild the registry in case we moved module files since the last deployment. A typical example is moving contrib modules from sites/all/modules/ to sites/all/modules/contrib/: without rebuilding the registry, your site will be broken and all following commands will fail.

drush vset maintenance_mode 1 sets the site to maintenance mode during the update.

drush updb -y runs all update hooks for contrib modules, core, and, importantly, your deployment module (we'll get back to that in a second).

drush cc all clears all caches, which can fix some problems during deployment.

On some projects, I have found that running drush cron at this point helps avoid hard-to-diagnose problems.

Finally, move your site out of maintenance mode: drush vset maintenance_mode 0.

hook_update_N()s

Our goal is for all our deployments (features, bug fixes, new modules...) to be channelled though hook_update_N()s, so that the incremental deployment procedure introduced above will trigger them. Simply, hook_update_N() are functions which are called only once for each environment.

Each environment tracks the last hook_update_N() called, and when drush updb -y is called, it checks the code for new hook_update_N() and runs them if necessary. (drush updb -y is the equivalent of visiting the update.php page, but the latter method is unsupported by the Dcycle procedure, because it requires managing PHP timeouts, which we don't want to do).

hook_update_N()s is the same tried-and-true mechanism used to update database schemas for Drupal core and contrib modules, so we are not introducing anything new.

Now let's see how a few common tasks can be accomplished the Dcycle way:

Example 1: enabling Javascript aggregation

Instead of fiddling with the production environment, leaving no trace of what you've done, here is an ideal workflow for enabling Javascript integration:

First, in your issue tracker, create an issue explaining why you want to enable aggregation, and take note of the issue number (for example #12345).

Next, figure out how to enable aggregation in code. In this case, a little reverse-engineering is required: on your local site, visit admin/config/development/performance and inspect the "Aggregate JavaScript files" checkbox, noting its name property: preprocess_js. This is likely to be a variable. You can confirm that it works by calling drush vset preprocess_js 1 and reloading admin/config/development/performance. Call drush vset preprocess_js 0 to turn it back off again. Many configuration pages work this way, but in some cases you'll need to work a bit more in order to figure out how to affect a change programmatically, which has the neat side effect of providing you a better understanding of how Drupal works.

Now, simply add the following code to a hook_update_N() in your deployment module's .install file:

/**
 * #12345: Enable javascript aggregation
 */
function example_deploy_update_7001() {
  variable_set('preprocess_js', 1);
  // you can also do this with Features and the Strongarm module.
}

Now, calling drush updb -y on any environment, including your local environment, should enable Javascript aggregation.

It is important to realize that hook_update_N()s are only called on environments where the deployment module is already in place, and not on new deployments. To make sure that new deployments and incremental deployments behave similarly, I call all my update hooks from my hook_install, as described in a previous post:

/**
 * Implements hook_install().
 *
 * See http://dcycleproject.org/node/43
 */
function example_deploy_install() {
  for ($i = 7001; $i < 8000; $i++) {
    $candidate = 'example_deploy_update_' . $i;
    if (function_exists($candidate)) {
      $candidate();
    }
  }
}

Once you are satisfied with your work, commit it to version control:

git add sites/all/modules/custom/example_deploy/example_deploy.install
git commit -am '#12345 Enabled javascript aggregation'
git push origin master

Now you can deploy this functionality to any other environment using the standard incremental deployment procedure, ideally after your continuous integration server has given you the green (or in the case of Jenkins, blue) light.

Example 2: changing a view

If we already have a feature which is a dependency of our deployment module, we can modify our view; update our features using the Features interface at admin/structure/features or using drush fu xyz_feature -y; then adding a new hook_update_N() to our deployment module:

/**
 * #12346: Change view to remove html tags from trimmed body
 */
function example_deploy_update_7002() {
  features_revert(array('xyz_feature' => array('views_view')));
}

In the above example, views_view is the machine name of the Features component affecting views. If you want to revert other components, make sure you're using the 2.x branch of Features, visit the page at admin/structure/features/xyz_feature/recreate (where xyz_feature is the machine name of your feature), and you'll find the machine names of each component next to its human name (for example node for content types, filter for text formats, etc.).

Example 3: changing the default theme

Say we create a new default theme xyz and want to enable it:

/**
 * #12347: New theme for the site
 */
function example_deploy_update_7003() {
  theme_enable(array('xyz'));
  variable_set('theme_default', 'xyz');
}

Example 4: adding and removing modules

I normally remove toolbar on all my sites and put admin_menu's admin_menu_toolbar instead. To deploy the change, add admin_menu to sites/*/modules/contrib and add the following code to your deployment module:

/**
 * #12348: Add a drop-down menu instead of the default menu for admins
 */
function example_deploy_update_7004() {
  // make sure admin_menu has been downloaded and added to your git repo,
  // or this will fail.
  module_enable(array('admin_menu_toolbar'));
  module_disable(array('toolbar'));
}

Don't change production directly

Of course, nothing prevents clueless users from modifying views, modules and settings on the production site directly, so I like to add hook_requirements() to perform certain checks on each environment: for example, if Javascript aggregation is turned off, you might see a red line on admin/reports/status saying "This site is designed to use Javascript aggregation, please turn it back on". You might also check that all your Features are not overridden, that the right theme is on etc. If this technique is used correctly, when a bug is reported on the production site, the admin/reports/status page will let you know if any settings on the production site are not what you intended, and what your automated tests expect.

Next steps: automated testing and continuous integration

Now that everything we do is in version control, we no longer need to clone databases, except in some very limited circumstances. We can always fire up a new environment and add dummy content for development or testing; and, provided we're using the same commit and the same operating system and version of PHP, etc., we're sure to always get the same result (which is not the case with database cloning).

Specifically, I normally add a .test file in my deployment module which enables the deployment module on a test environment, and runs tests to make sure things are working as expected.

Once that is done, it becomes easy to create a Jenkins continuous integration job to monitor the master branch, and confirm that a new environment can be created and simpletests pass.

Tags: 

Comments

Great implementation of a deployment workflow based on the mantras of "Don't touch production" and "Sinlge source of the truth - in a version-controlled repository".
Taking "drush updb" to the extreme by making hook_update_N() the workhorse of an entire install, rather than individual modules is certainly an innovative approach.
Not taking away anything from your approach, I personally prefer the UI of Features+Strongarm and exporting that to a module to go into Git over coding my own variable_set().
Re: "nothing prevents clueless users from modifying views, modules and settings on the production site directly"... one company I worked for which was paranoid about security solved this simply by stripping their sites from the login page, i.e. removing /user from the menu. Works if your site is brochure-ware. If the site DOES need a login, then maybe roles with "dangerous" permissions could be programmatically removed as part of deploying to production.

Great process ! I try to implement something like that and i find yours very interesting.
I wonder if you have a fallback process if your implementation in hook_update_N failed in production for example ? Thx

Thanks @Mk. In practice I have rarely come across problems in production with this approach, especially if done as such from the project get-go. Here are two instances where problems have occurred and how I fixed them:

The first was when trying to install a module. For some reason the install hook of the module was not being called, so when calling cron later on I got a database table missing error, but only in production. It turned out that a previous (1.x) version of the module had been installed, and disabled (but non uninstalled) on production. Because there was no reliable upgrade path to the 2.x version which I was using, the schema was not installed correctly. My solution was to increment the update hook and add a line specifically uninstalling the module before reinstalling it, then running updb again on prod, and it worked.

I have also seen problems with Features revert in update hooks, but this happens rarely. In this case, I meddled with the production database directly for my update hook to work. Even then, if I had wanted to things cleanly, I imagine it would have been possible to do all those operations in a subsequent update hook.

Having used this method on all my projects for almost two years, I have had less than ten such weird problems, so I would say the method is quite robust. So, to answer your question, my fallback is to back up the production database before updating it; and if it fails, understanding the problem and either correcting it manually or with a subsequent update hook.

Cheers,

Albert.

Thanks Albert,

I set up my deployment script and i'll share discoveries, joys, disappointment with you.

Matthieu

Thanks for a great article. It has inspired me to get a bit more discipline with updates.

One thing, I wonder if it would be better to drush fra the production site on each deployment rather than putting specific features_revert calls in the update functions. Two reasons:

1. If your configuration is all exported to code correctly, reverting all features should be fine rather than targeting a specific feature.
2. With a lot of deployments your .install file is bound to get a long list of update functions that do nothing but revert a feature. This gets a bit tedious to write, also it means if the module is installed, all those reverts will be run.

I'm reverting features on deployment via acquia cloud deploy hooks with great success.

Thanks for your thoughts, Phil. Just to remind everyone, drush fra reverts all your features on a given environment. In my opinion, the only really important thing is that the exact same script be run for updates every time you incrementally deploy. Adding drush fra can be a good addition to that list of steps for many projects, even though drush updb is necessary anyway because there are some things features just can't do.

I guess in setting it up the way I did, I wanted to have fine control over what exactly was happening in each deployment. In some cases, drush fra can be finicky. For example I often get this issue documented at drupal.org/node/1822278, which sometimes slows down my deployment procedure and never happens if I do it in hook_update_n()s. I have had other issues with features using Ubercart base types causing conflicts if they were reverted, even though I hadn't changed anything.

On the other hand, you have to remember to revert specific components in the hook_update_n() with my procedure, and if you forget it might cause confusion.

Bottom line: if you have a standardized procedure that works for you, great. If we get together as a community and set up standards that we all use, all the better. Should drush fra be part of those standards? I'm not sure yet, I'd love the discussion to continue.

Running drush cc all immediately before drush fra -y should solve the drush features-revert-all wrongly tells me features is not enabled problem.

I'm actually just starting out with the Drupal + Git + Drush trio, and I've been scratching my head on how to do and organize incremental database changes.

This will be my guideline.

Awesomely put together!

Thank you!

I'm getting into code-based development and found your post, with its clear examples, a godsend. I'm puzzled as to why you start numbering hook_update_n calls at 7001; the Drupal handbook page says "2 digits for sequential counting, starting with 00."

Not criticizing, just curious.

Although I've never actually done a major-version port, from what I understand from the API documentation:

  • 7000 is reserved for upgrading your database during a major-version (6.x - 7.x) upgrade.
  • 7100- 7199 is reserved for upgrading within a minor version (7.x-1.x)
  • 7200-7299 is reserved for the following minor version (7.x-2.x)

This is fine for developing modules, but when you're developing a site with a site deployment module, you typically don't have minor versions, just a master branch. Still, I never actually tried starting at 7000 as I was always under the impression that 7000 is run only when migrating from Drupal 6 to Drupal 7. (If I'm wrong, let me kwow!)

The following workflow has worked well for me:

  • I don't define 7000, signalling that there is no 6.x of my site or, if there is, I did not define an upgrade path.

  • I start at 7001

  • In my site deployment modules there is nothing special about 7100, 7200, etc.

  • for a medium-sized project, I normally end the project at about 7100 to 7250 and have found no ill-effects.

  • if a number is missing, Drupal will just move on to the next one during database update. For example, It's OK to have 7040, not 7041 to 7045, and the next update being 7046. This is useful during development when you made a mistake in an update and it does nothing, you can fix it by incrementing its number. Also, if you are using hook_update_n to revert your features, you don't need to revert the same feature twice, so you can safely delete previous update containing the same features_revert code.

Add new comment

Markdown

  • Quick Tips:
    • Two or more spaces at a line's end = Line break
    • Double returns = Paragraph
    • *Single asterisks* or _single underscores_ = Emphasis
    • **Double** or __double__ = Strong
    • This is [a link](http://the.link.example.com "The optional title text")
    For complete details on the Markdown syntax, see the Markdown documentation and Markdown Extra documentation for tables, footnotes, and more.

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <h3>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.