Beware Sourceless Moves!

A few months back we had one of our servers go down, fortunately it "only" had about 350 users on it. Well I thought what a great opportunity to test out the sourceless move option to our hot standby server in our Colocation site.

I moved a few users at a time to be sure it worked, and it only seemed to take a minute or so per user. So I got some confidence and started moving 10-20 people at a time. (I didn't get confident enough to bulk move more than that) All this time I was watching the logs to see when the accounts completed the move process to gauge how fast it was going. Pretty nice, actually, the move automatically updates the service books on the device with the new SRP ID, so no user intervention required, in fact it truly is seamless. Nice.

Well after a couple of hours all the accounts were moved over and I focused on rebuilding the original server. Then after awhile I noticed something odd - there were some error messages in the log I was not familiar with:

[NoteID] was not found in any of the 1 redirected folders

Well actually I have seen these before but not so many of them. After a LOT of investigation, I discovered that Sent Items were not being redirected to *any* of the devices that had been moved to the new server.

Although I had replicas of all the user's State DB's on the Colo BES server, I had assumed that I did not a backup of the BlackBerryProfiles.nsf database. Upon each move, the new BES would pull info out of SQL and create a new Profile entry in this empty database. "Great", I thought, "that creates a fresh clean profile entry, and besides, we are SQL all the way now, so these profile entries aren't really used."

Ummm.... not quite.

Creating a fresh profile DB entry for each user misses some critical information that, apparently - even in this age of the SQL DB as the master and core of our BES environment - exists only in the profile DB entries.

I will name names here. In particular, the following fields:

1) DeviceCapabilities
2) RedirectSelectedFolders

First of all, DeviceCapabilities is set to null upon a sourceless move, and RedirectSelectedFolders doesn't even exist!

What are the ramifications of this? Well for DeviceCapabilities, the BES server will note that it is set to nothing and assume that it is a pre-4.0 device, and thus will not reconcile the Sent Items from Notes onto the device, and will throw the error I noted above.

For RedirectSelectedFolders, this would be created and set to "1" if someone went into their subfolders and marked selected ones for redirection to the device. If this field doesn't exist, then the BES assumes naturally that they don't have any subfolders to redirect.

So the "seamless" sourceless moves result in all devices not getting Sent Items or any custom redirected subfolders. Not good.

Of course, reactivating the device fixes this stuff. But was I about to call 350 users and reactivate them all? Nope.

In order to fix this from the server side (which is where I prefer to be), I had to get a backup copy of the BlackBerryProfiles.nsf DB from the downed server, and script an agent to populate the fields in the new Profiles BD. Although partially scripted, this was not automated, so I ran the script for each account, copying and pasting in the long string in the DeviceCapabilities field which the script then set in the new Profiles db entry.

When these fields matched what was in the previous Profiles db, then magically people began getting their Sent Items and subfolders redirected as normal. Not a fun day for me though.

Of course I called RIM to ask about this "bug" and was told that they knew about this behavior but it really wasn't planned on being fixed.

The upshot is.... never allow the BlackBerryProfiles.nsf database to be recreated, either by deleting it yourself or doing a sourceless move to a new server. You will lose valuable info in each document. Instead, in addition to the backup replicas you have of the state DB's, have a backup replica of this DB as well, so that in an emergency you can copy and paste the profiles into the database on the backup server, so they don't get autocreated and inherit all the fields with valuable goodies in them.

One final note, this only applies to sourceless moves - if you have the original BES server up and available during an account move, the destination server will pull all the information from the original profile document into it's database cleanly.