Building your own platform on top of Drupal for managing many sites.
(Drupal folks I would love to hear how this is similar or different than your workflow. Please leave a comment).
Using Drupal's multi-site features, installation profiles and Drush, features and context modules, as well as a version control repository like Subversion, you can build your own fine-tuned platform to deploy and host multiple sites with similar features while still retaining the ability to customize each particular site as needed.
Drupal has core and modules updates that cover bug fixes, security and new functionality. Additionally, site owners and developers will want to add new functionality from over time. All of this is quite normal and usually fairly manageable for one Web site. However, once you find yourself maintaining and developing several sites scattered across several servers, a single security update can make you cringe.
Why? Let's take a look.
The one and only server approach
Here we have a single server hosting a single Drupal site. Let's say a security update comes along. OK - so the site admin copies the new files via FTP, or better yet SSH, to the Web server, logs in as the administrator and runs update.php. As long as everything goes well, all is good. Usually, everything does go well, but on occasion we make a mistake or some other problem rears its ugly head. It would be nice to know ahead of time if there was going to be a problem, but in this scenario, that's not likely.
One development and one production server
So lets rectify this somewhat by having the site admin make a local copy of the site on his/her computer. The site admin can even pull a copy of the live site's database and use it on his local install to simulate the sever environment. This looks like this:
This way the site admin can test the updates on the local development server before deploying them to the production Web server. If anything goes wrong, the site admin can sort out the problem before they go live.
The draw back here is that there are 2 sets of updates to run and this is twice the work. We could already benefit from putting Drupal into a code repository, but lets take this a step further first.
Four Local and four Production Servers
Now lets assume our drupal admin finds himself/herself in charge of 4 sites on 4 servers.
When a security update roles along, the site admin now has to apply the update 4 times on the local server and on 4 times on 4 production servers.
We are skipping over the fact that many sites deploy local, staging and production servers. So our 8-headed monster could easily become twelve.
So congratulations, you have created a nearly unmaintainable monster!
Enter Subversion
Subversion is a version control repository (please note that you can substitute CVS, Bazaar, Git or your favorite version control system here). The basic idea is all the files in your Drupal install go into the repository that lives somewhere out on the Interent. All updates to existing files and all new files are sent to the repository via the local development server. The production server can have a copy of the same repository. Updating to the latest version becomes a matter of running 'svn update'.
This significantly simplifies the file management aspect of maintaining one or more Drupal sites. Plus we can add Drupal core and all of our typically used modules to the repository so setting up a new site is a simple matter of checking out a copy of your repository. This also solves the problem of setting up a new Drupal site which requires downloading Drupal core, and all the typical modules individually like views, cck, pathauto, token, etc... Its not unusual to use 20 or more modules on a single Drupal site. Downloading them one by one for each new site is not particularily efficient. With a Subversion repository we have this ready to go and can checkout a copy with 'svn checkout' at any time.
Still, we can optimize our setup even more by using Drupal's multi-site feature.
Multi-site Drupal Installations
Drupal lets us run multiple sites on a single code base. If we take a look a Drupal's directory structure, we see a folder called sites. Inside this folder we see sub-folders called default, and all. If we just want to deploy a single site. All of our modules and themes can go in the default folder. However, we can create new folders for each site inside the sites folder. In the image below I have four Drupal sites running on one installation of Drupal. The image shows my sites folder on the production server for 4 drupal sites.
So you can see mnitcareers.org, mnitworkforce.org, secure360.org and strategicit.org all run on a single installation (strategicit.local and mnitcareers.local all simply aliases or symbolic links and .svn is a folder Subversion uses to operate).
To set this up we simply point our Web server and DNS to the root Drupal folder (usually in the Web server root directory) for each site (that's right they all point to the same place). Then Drupal reads the HTTP header and directs each request to the appropriate site.
All of our shared modules and themes can go into the all sub-folder to make them available to all sites. The great part here is each site can have its own modules and themes for additional customization as needed.
So we have moved from from managing 8 (12 if we use staging sites) instances of Drupal without a code repository to 2 instances (3 with a staging sites). This is a huge improvement!
Let's review our security update workflow now. We add the updated files to our Subversion repository via our local development server, run update.php for each site and make sure everything is in good order. Then we log into the server via SSH and run 'svn update' to get the new files. Then we run update.php on each site on the server.
Ah, we still have to run update.php for each site for a security release, bug fix or other update. This is somewhat unavoidable, but we can make it easier and quicker.
Enter Drush
Drush is a command line Php script that allows us to do Drupal maintenance quickly and efficiently. To update a Drupal install via Drush, we simply navigate to the sites folder for a given site and run 'drush updatedb' - no waiting for browser pages to load, no logging into Drupal. This can be done by logging into the server via SSH. We still need to run it it once for each site, but it runs very quickly. Additionally, seasoned Unix command line users can make a script to run the update for each site in the multi-site install. So it is possible to get to the Holy Grail of Drupal maintenance.
Drush also has other tricks up its sleeve like downloading core and module updates. For example 'drush dl views' downloads the latest version of the views module.
So looking at our security update now, the site admin can download the updates with Drush. Apply all updates locally with a Drush-based script. Assuming everything looks good, she can commit changes to the Subversion repository, run the Drush-based update script on the server.
Code Vs. Database
With maintenance out of the way we can concentrate on building out our Drupal site. This usually includes configuring Drupal core, configuring modules blocks, creating views and content types as well as building a custom theme. Themes are file based so they can work nicely in a repository . Modules and Drupal core however store configuration information in a the site's database. This isn't particularly problematic at the start of a project as a copy of the production ready site's database can be imported to the production server, but after a site has been running and content is added daily, this becomes very difficult to manage. Essentially local and production databases need to be merged.
Drupal Features and Context modules solve this problem by exporting configuration of content types, views and more to files that are essentially Drupal modules. Then these files can be submitted to the repository and deployed on a production server.
Installation Profiles and Building Your Own Platform
All of this effort allows us to leverage Drupal into our own end to end platform. One additional ingredient, an Installation Profile, lets us take this a step further. Imagine we want to build a series of sites with common features, say forums, user profiles, and a few additional social networking features. We can build this out once and turn it into an installation of Drupal with all of our feature requirements and deploy the installation as many times as we like. The install profile simply shows up as an option during installation. Roll-out becomes as easy as checking the right checkbox during the installation. Then you can simply concentrate on making each site visually different via the theme and design!
Drupal
admin's blog
Add new comment
Environments Module
Submitted by Ben (not verified) on Fri, 2009-07-24 09:15.
I recently released a module called Environments that helps manage Drupal sites/hosts "two-dimensionally", as sites on each build and environments (production/dev/local). Check it out: drupal.org/project/envts.
- Ben B
reply
Thanks Ben
Submitted by admin on Sun, 2009-07-26 20:59.
Looking forward to checking it out!
reply
Elaborate more about using SVN
Submitted by Anonymous (not verified) on Fri, 2009-07-24 11:26.
Hi there,
Great article, but can you elaborate more about how to use subversion to manage your drupal sites. Many sites say to just put Drupal into subversion, but very few explain the step by step details of the workflow. Maybe this would be a good article? You could show how to do this with Windows and Tortoise SVN with lots of screen shots.
How do you handle tracking the changes between Drupal core updates in your repository? You can't just copy over the top of the previous version and commit.
Do you create a separate repo for Drupal,a separate repo for modules, and a separate repo for each site?
Just some questions that came to mind.
reply
Here is a rough breakout
Submitted by admin on Sun, 2009-07-26 21:04.
Drupal core and all the modules I typically use in one repository.
Then each site gets it own repository.
When I add new files or update existing ones, I run:
svn status
svn add file 1, file 2 .... (if needed)
and svn commit -m 'comment'
This sends the new files to a reposiotry in the cloud.
Then on the production server its just :
svn update
Have a look here for a more detailed explanation and video:
http://gotdrupal.com/videos/drupal-cvs-svn-setup
reply
hosting clients' websites
Submitted by Aaron (not verified) on Fri, 2009-07-24 11:42.
Nice article - I would love to manage only one drupal core... But for me this methodology breaks down with respect to hosting for multiple clients.
Where I'm at now, I don't really have the capacity to administer and manage my own server, so my clients all have their own hosting accounts. This eliminates the middleman (me) which is good for the client and good for my bookkeeping, invoicing, kernel updates... etc.
It looks like all your clients not only share hosting, but share a drupal core to boot, so my questions are:
Are you re-selling hosting? Are you hosting in-house? Something in between?
Have you run into any issues with one high-traffic site slowing down all the others?
How transparent do you make the shared setup to your clients?
Do they know they're all sharing the same code base?
reply
Good questions
Submitted by admin on Sun, 2009-07-26 21:14.
I host an primarily 2 places and a few clients have separate hosting.
The 2 primary sites have one install. In one case (at work) we have a series of related sites so this makes perfect sense.
In the other, I just let people know but to be honest - no one has really seemed to care as long as the Web site is working. Most of these really aren't high traffic sites - so its not much of an issue.
In the other cases I use a unix tool called sitecopy. This is a sort of automated ftp program that keeps track of which files are already on the server and which ones need to be updated. You can set it to ignore all the svn files. I do a seperate snv checkout so as to not mix all the files from seperate sites.
Though I think I will charge an extra fee to clients who need separate hosting from now on - just because I have to take the time to set up Drupal for them and they typically don't have SSH access so I have to do updates manually instead of with SVN/Drush.
reply
My thoughts
Submitted by Kaerast (not verified) on Sat, 2009-07-25 18:30.
Drush is fantastic, but isn't quite the holy grail. It can't cope with updates to Drupal core (for that I created DUPG), it doesn't understand any vcs system other than SVN, it gets confused when PHP4 is still available to it, and it has to be told what multisites you have.
If you are running everything off of a single svn repository, so that all modules you have ever used are available to all sites you have ever created, then you end up with a huge list of modules and themes available. That in turn uses up lots more memory than it should do. I've always used this method of managing multiple Drupal sites myself, but am starting to move away towards using Drush more and more.
I've never created Drupal install profiles, and the only time I've seen it done well is Open Atrium. The rest of the time, especially if it's only for internal use, it's quicker and easier to write a bash script which uses Drush to download and enable all the modules and themes you want.
Finally, it's worth mentioning Drupal Migraine. I haven't had time to test it myself yet, but it sounds like it should solve the syncing between development and live sites quite nicely.
reply
Well don't agree entirely
Submitted by admin on Sun, 2009-07-26 20:57.
Kaerast, I keep all my typical modules and drupal core files in one repository.
Then each site I do is in another - this includes the sites/www.thissite.com folder in which I have the theme and modules used only on that site. So each site can have its on site-specific modules and keeps everything cleaner.
reply
Aegir
Submitted by askibinski (not verified) on Tue, 2009-07-28 10:41.
What about Aegir?
We don't use it (yet), but I saw a demonstration at a recent DrupalJam about this hosting system (also uses drush) and it looks promising. Especially if you have a lot of similar featured sites to manage/deploy.
reply
Haven't tried Aegir yet
Submitted by admin on Mon, 2009-08-24 08:39.
but it is on my to do list
reply
multi-site to single site
Submitted by Eric (not verified) on Fri, 2009-08-14 07:25.
Do you have any experience moving from a multi-site installation to a single installation? I'm starting to use multi-sites for development, but I'm not hosting most of the sites, so I need to move them to their own hosts after building them. But the references to the multi-site folder need to be changed before moving the database. I'm just wondering what's the best way to handle that.
reply
Sure
Submitted by admin on Mon, 2009-08-24 08:39.
Just keep it the same except only move the site folder for the site in question, leave the others on your machine. I also find it best to switch back to Garland before exporting the database and switching the theme back to the original theme after the transfer is done. No surprises that way.
Backup and migrate module works great for importing and exporting the database.
reply
Features & Content
Submitted by Nader (not verified) on Thu, 2009-08-27 15:39.
Hey Geoff, thanks for this. There is some great info here.
So my company is going through the same process. We're providing a service based on Drupal that we will offer to multiple clients. So we have individual developer environments, each with their own db, a test server, a staging server/db per client and a production server/db per client.
So certain changes we want to push to all clients, like new functionality, new modules, changes to certain modules, etc. Other changes, like menus, taxonomy terms, etc we do not want to push to everyone.
Can you please explain in a little more detail how you use the Features & Content modules to achieve this? I'm also looking into the Database Scripts project which looks like it might help out in this area.
I'm also happy to take this offline, if that's easier.
Thanks again for the great article,
Nader
reply
Checkout the Features and Context modules screen cast
Submitted by admin on Tue, 2009-09-15 08:46.
over at drupal.org/project/features
reply
Drush and SVN Clarification
Submitted by JamieT (not verified) on Fri, 2009-09-04 09:11.
Hi - thanks for the write up I'm sure this will help me setup my own workflow. Just wanted to clarify a couple of points.
1. Which environments do you have Drush installed on (Prod, SVN Rep, Local Dev)?
2. Would the workflow be similar to below:
-local dev drush update to update the files
-local dev updatedb to update the local dev db
-check all is ok then svn commit to update the files in the rep.
-Then you would svn update on prod to update files
-and drush updatedb to get the prod db updated
Is that correct?
Thanks in advance.
JamieT
reply
Drush and SVN Clarification
Submitted by admin on Tue, 2009-09-15 08:46.
1. usually all but I primarily use Drush in the dev environment
2. Yes that is exactly the workflow
reply
No comments:
Post a Comment