ColdFusion Publishing on Demand from Subversion to Load Balanced Coldfusion Servers

As with previous blogs, this entry is a note-to-self that I thought others might find useful.

[Edit: Jan 16, 2012 Good grief... I wrote this in 2009 and left it in draft because I didn't complete the last 5% of the post. *shakes head*]

This is our web production environment.

We develop on our local machines and stage on a common internal server in California. Code revisions are stored on a subversion server (VisualSVN Server) located in our DMZ. Our production environment consists of two load-balanced ColdFusion servers hosted by Rackspace in Dallas, TX.

Our publication process is simple:

1. Export from svn to a temp folder on Prod1.
2. Backup Existing Production copy, and rename temp folder to production folder. 
3. Synchronize Prod1 with Prod2.

The tedium of doing this process manually pushed me to automate. I am not familiar with Ant, so I turned to DOS batch files. 

Step1: Export from Subversion

Slik Subversion: Windows Command-Line SVN

Working with batch files means finding tools that support command-line operation. I found Slik Subversion, a free full-featured windows command line client for svn. The help file was very useful in building the export command I needed:

C:\Program Files\SlikSvn\bin>svn help export
export: Create an unversioned copy of a tree.
usage: 1. export [-r REV] URL[@PEGREV] [PATH]
       2. export [-r REV] PATH1[@PEGREV] [PATH2]

  1. Exports a clean directory tree from the repository specified by
     URL, at revision REV if it is given, otherwise at HEAD, into
     PATH. If PATH is omitted, the last component of the URL is used
     for the local directory name.

  2. Exports a clean directory tree from the working copy specified by
     PATH1, at revision REV if it is given, otherwise at WORKING, into
     PATH2.  If PATH2 is omitted, the last component of the PATH1 is used
     for the local directory name. If REV is not specified, all local
     changes will be preserved.  Files not under version control will
     not be copied.

  If specified, PEGREV determines in which revision the target is first looked up.

Valid options:
  -r [--revision] ARG      : ARG (some commands also take ARG1:ARG2 range)
                             A revision argument can be one of:
                                NUMBER       revision number
                                '{' DATE '}' revision at start of the date
                                'HEAD'       latest in repository
                                'BASE'       base rev of item's working copy
                                'COMMITTED'  last commit at or before BASE
                                'PREV'       revision just before COMMITTED
  -q [--quiet]             : print nothing, or only summary information
  -N [--non-recursive]     : obsolete; try --depth=files or --depth=immediates
  --depth ARG              : limit operation by depth ARG ('empty', 'files',
                            'immediates', or 'infinity')
  --force                  : force operation to run
  --native-eol ARG         : use a different EOL marker than the standard
                             system marker for files with the svn:eol-style
                             property set to 'native'.
                             ARG may be one of 'LF', 'CR', 'CRLF'
  --ignore-externals       : ignore externals definitions

Global options:
  --username ARG           : specify a username ARG
  --password ARG           : specify a password ARG
  --no-auth-cache          : do not cache authentication tokens
  --non-interactive        : do no interactive prompting
  --trust-server-cert      : accept unknown SSL server certificates without
                             prompting (but only with '--non-interactive')
  --config-dir ARG         : read user configuration files from directory ARG
  --config-option ARG      : set user configuration option in the format:
                                 FILE:SECTION:OPTION=[VALUE]
                             For example:
                                 servers:global:http-library=serf

Since this code will be executed by SYSTEM, I need to explicitly pass the username and password. I want to make sure svn doesn't throw any user prompts (--non-interactive) and it should overwrite anything that is already there (--force). (In truth, the --force isn't necessary since I wipe out the temp folder before export, but it eliminated a point of failure while I was building this process.)

"C:\Program Files\SlikSvn\bin\svn.exe" export --force --username cfauto --password somepass --non-interactive "http://subversion.mysite.com/svn/main/webapp/trunk" "D:\inetpub\webroot\_microsites\thiswebapp_

Tighten Security

To reduce security risk created by having an exposed plain-text password, I created a user, cfauto, on the VisualSVN Server that has read-only rights to the thiswebapp repository and nothing else.

Escaping % in DOS

When I was testing the svn command-line export above, I used my personal credentials. I couldn't figure out why it wasn't working until I noticed the percent symbols in my password (some%xyz%pass). In DOS, variables are bracketed by percent symbols (e.g. %variablename%). When a DOS variable is undefined, it is treated as a blank rather than throwing an error, so in my password, %xyz% was treated as a variable, causing some%xyz%pass to be rendered as somepass at execution time. To escape the percent symbol in DOS, use %%. Rewriting my password as some%%xyz%%pass worked perfectly.

Step 2: Move into Production

Best Practices: Backup and Swap

This is rudimentary and (some might say) common sense, but I think it is worth mentioning since I had to figure this out on my own when I started years ago. When you deploy to production:

1. Deploy to a temp folder xxxx_
2. Rename your existing production folder xxxx to xxxx-old
3. Rename your temp folder xxxx_ to the production folder name xxxx

Exporting a project takes time. If you wipe out your existing production folder, then export directly to production, you risk bizarre errors as users visit your web app before all of the files are in place. Renaming folders is effectively instantaneous.
I like having the existing production folder renamed to xxxx-old. If there is a problem with deployment and you need to roll-back, all you have to do is rename xxxx-old to xxxx and you're done. You don't have to wait for a previous-version export from subversion.

Step 3: Synchronize Secondary Servers with Primary

Backup and Swap in a Load Balanced Environment (optimal scenario)

When working in a load-balanced environment, optimally, you should:

1. Export to a temp folder
2. Synchronize the temp folder to all servers
3. Trigger Backup and Swap on all servers simultaneously

In practice this is tricky. Whatever mechanism you use, you have to be sure that the temp folder has copied completely to all servers before triggering the backup and swap routine. The best solution I can think of is a scheduled process on each machine that performs the backup and swap steps on all temp folders (marked with an underscore). Running this "optimal" process manually could be accomplished with remote execution, but I haven't thought this through. (A good topic for another post.)

Sync Short-Cut for a non-HA Environment

Since our apps don't have a high-availability requirement, I just let our sync software work directly with the production environment on Prod2 (slave). Instead of copying the temp files over, I do the backup and swap on Prod1 and copy the finished product to Prod2. Prod1 and Prod2 are connected by a gigabit LAN, so copying from Prod1 to Prod2 is orders of magnitude faster than deploying from a repository located in another state. The apps we have typically sync in 5 seconds or less.

FreeFileSync: GUI File Synchronization for Windows (with command-line controls)

FreeFileSync is a fantastic free tool for keeping files in sync on a LAN. (I use version 3.3, and it does not support FTP at this time.) Two cautionary notes with FreeFileSync:

1. I have had problems when syncing large files. We have several videos that are 150mb and larger. Occasionally, the secondary copies of large files will have some corruption. The file size is correct, but the videos will stop playing in the middle. If I copy the file again using windows explorer, it works.
2. Upgrading can break your rules.

The interface for FreeFileSync is easy and intuitive. It supports UNC paths, and the creation of batch jobs. The project is very active and new features added all the time. As of this writing, the most recent version is 3.21.

Scheduling the Sync

(To Be Continued...)