How to Recover* from a Total Loss of BESMgmt SQL database (* with caveats...)

The BESMgmt SQL database is the heart of the Blackberry Enterprise environment. Without it, BES servers will continue to deliver mail but no meaningful changes can be made, or BES services restarted, until it is back up.

The only piece of the BES environment that is out of my control is the database, which is hosted and managed by our SQL team. Although I have full confidence in their replication, backup, and recovery plans, I still need to plan for the (hopefully very remote) possibility that the SQL db might go bye-bye someday.

Until now, this worst case scenario resulted in an "oh well, everybody will have to reactivate their devices" DR plan. I dread the day I have to say this to the CIO!

Recently, however I got wind of the idea of using the nbesmigration.exe utility for DR purposes. This utility was originally designed to populate user accounts in the 4.0 SQL database from a version 2.2 profiles database. Since the 4.x environment still maintains a BlackBerryProfiles notes database on each server, the tool could theoretically be used to re-create the user accounts from these profile entries into a brand new SQL database.

While running the tool in testing, I noticed the following text as part of the built-in help:

BlackBerry Enterprise Server(c) for Lotus Domino(c) Migration Tool, Version 4.1
Copyright (c) Research In Motion, Ltd. 2004, 2006. All rights reserved.
Modification date: May 8 2007

Usage: NBESMigration.exe [options]

This tool can be used to migrate entries from BESD 2.2.x Profiles database to BESMgmt database (UserConfig table). For migration, create the database first using CreateDB, create a server entry in ServerConfig, and run this tool from your BESD server's Domino directory.

This tool can also used for disaster recovery or fixup purposes on already installed BESD server (4.0 and up). Shut down the BES, and run this tool from your BESD server's Domino directory. The tool will check all user docs in Profiles DB, create entries, if needed, in BESMgmt DB (UserConfig table), and it will link entries using SQLID field.

From the text it appears that this tool does support a limited form of DR. Good news! So to test, I did the following in my test environment:

1. Detach the BES SQL db and drop all connections.

2. Stop BES server task and services.

3. Run the BlackBerry Congfiguration Utility and choose the Change Database option.

4. Enter a different database name, thus creating a brand new database.

5. Re-enter CAL and SRP information, but do not restart services.

6. From the tools subdirectory of the BES 4.1 installation share, copy the nbesmigration.exe executable to the c:\lotus\domino directory

7. Run the following command nbesmigration -d: "[SQL Server name]" "[SQL DB name]" -u: [SQL login] [password]. In my case this command line looked like this:

nbesmigration -d: "SQLTEST01" "BESMgmtTest" -u: besdbowner besdbpassword

The tool indicated that it found my one test account and migrated it successfully.

8. Restart the BES server task and services.

The server fired up and I was able to immediately send and receive messages, and do lookups from my device! So this is a great way to recover basic functionality in the case of a total loss of the SQL db.

Keep in mind that you need some information documented and handy to provide to the configuration wizard:

• CAL licenses
• SRP ID's
• SRP Authorization keys

Now, for the caveats:

• IT Policies are reset to defaults (!)
  
• PIM Sync Mappings are reset to defaults
• Email Settings are reset to defaults
• Probably a bunch of other stuff is gone or reset to defaults

Although some of the other stuff is fixable, I was unable to successfully apply the default IT policy or a newly created IT policy in the new enviroment, even when it was named the same as the old IT policy. It always came back with an error. On the other hand, when I switched DB's back to the original, the original IT policy was able to be re-applied successfully.

The upshot is that this is a last-last-resort option to get mail flowing to the devices without having everyone reactivate. If you can later restore the original SQL DB and switch back to it, then IT Policies, PIM Sync settings, etc, can be restored back to their original good state.

If you can never get the original DB back, however, then you might have to move forward and find some creative solutions to the problems above. If the solution does eventually require handheld reactivation, well then at least you can control and stage it on your terms, without having hundreds of people screaming at you to get their BlackBerry email fixed!