BlackBerry Enterprise Server on Domino Discussion

Moving On...

2009-03-06T06:18:00.000-08:00

Well today is my last day working on Domino systems, so I guess it is the last day for this blog as well. I have had fun probing the inner workings of Domino as it relates to BES, however all good things must come to an end, and I will soon be going back to the dark side. Adieu!

Software Configuration Push Problems Wrapup

2009-01-08T13:22:00.000-08:00

Well here is the upshot of pushing an application to ~1500 Blackberrys: About 50 of them will not receive the app for a variety of reasons. (Which is actually not too bad considering the complexity and fragility of all the pieces involved in a wireless software deployment!)

First, check the Last Contact Time for the user's BES account. I have about 8 of these guys who haven't been heard from in weeks or months, they are probably on maternity / paternity leave or have been fired but not deleted from the BES. Ignore these guys for now.

Second, ensure there is enough memory on the device via the BES Manager. If the device reports 1MB free then it probably won't install the app. If the last date status was reported is way in the past, the device probably doesn't have enough memory to even report daily status anymore.

Then, check that the device's Hardware information is reported in the BES Manager. Some devices only have there "Secured Boot ROM" entry there, which means the hardware is not reporting correctly, and is reporting a hardware ID of 0x00000000. The BES doesn't see this number in the device.xml and won't push the app to it. This happens alot with Verizon devices, especially the 8300 models, but I have seen a few with T-Mobile 8700's as well. In this case the only thing I have discovered that fixes it is to upgrade to a later OS, such as 4.5.

If memory is plentiful and hardware info is there (and of course your device.xml is updated), then I have found that there may be a mismatch between the Applications / Modules reported by BES Manager and what is actually on the device. Many times the BES Manager will report multiple modules with different version numbers, but the device is actually just fine. In this case we need to clear the BES Manager device info. The best way to do this is to go into the PIM Sync Options of the user and set the Automatic BlackBerry Device Management Enabled setting to False, clicking OK, waiting 1 minute, then going back in and setting it to True. This will clear the data from the SQL tables, and when set back to True, will request that the device repopulate the information with fresh data. When this is done, after a few minutes, often the software push will begin to work.

On some devices I set the above setting back to True and I never get back any device information! In this case the Desktop (SYNC) service book on the device seems to be corrupted, and having the user delete it and undelete it restores everything just fine. Then the push works.

Please note that deleting the user's BES account, recreating it, wiping the device, and reactivating will most likely fix all the above issues. However in our environment it is imperative that our attorneys have as little down time as possible, so fixing these issues behind the scenes is a lifesaver.

Hope this little learning experience helps you all out there as much as it has me.

Software Configuration Push Problems cont'd...

2008-12-23T07:13:00.000-08:00

Another issue I discovered is the following:

1. I audit all of our devices to ensure the memory is above 3MB before pushing the software, to avoid causing their devices to delete all their mail. How to do this is another post in itself!

2. We send these users emails on how to clean up the device.

3. I got some emails back saying they had plenty of memory free already!

4. I checked and found that the Device information had not been updated in 6 months or more! I was working with old data for some users!

Here is my theory:

Once you get to a very small amount of memory on the BB, the Handheld Agent will stop reporting back to the server it's details on a daily basis. Fortunately there was an easy fix for most of these guys, and that is to set "Automatic BlackBerry Device Management Enabled" to FALSE, wait a few minutes, then set it back to true. What this does is delete all the device information for that user from the SQL database, then when you turn it back on it refreshes it directly from the device. If there is enough memory available you will get new info. If memory is still low you will not get any info back.

Interesting side note: The refreshed device information will only reflect when the Handheld Agent last collected the data, not the "right now" state. So if you get todays info it may be from 5 or 7AM, not from 11AM when you changed the settings.

Software Configuration Push Problems (SCPP's)

2008-12-23T07:05:00.001-08:00

Ok here I am again after a long bout of laziness. Anyway I am working on deploying a piece of software to all our devices and am running into fun issues. Here is one of them:

8330 Curves on Verizon running 4.2 or 4.3 OS, for some reason, do not publish their hardware details to the BES. In fact they don't even publish it if you are connected over USB to the device via BlackBerry Manager. The Hardware ID is just 0x00000000.

This is a problem because when you push software it checks the device model against the device.xml file in the AppLoader directory above the actual software package. If it doesn't find it then you get an error in the POLC log about HHCM_DEVICE_NOT_SUPPORTED and to Update your device.xml.

The issue of course is not the device.xml, but that the Hardware ID is not there for this device.

I found that upgrading from 4.2 or 4.3 OS to 4.5 will publish all the gory hardware details and software pushes will work again.

I wonder if it is stupid Verizon wanting to hide the hardware from everyone but themselves (like they do with GPS) or just an bug in their version of OS 4.2 / 4.3. Either way seems it is resolved in 4.5.

ITAdminQueue - EXPOSED!

2008-05-07T09:37:00.000-07:00

A pretty cool new feature appeared in 4.1 SP5, resulting in a new tab called Software Configuration Status within the BlackBerry Manager application:

This tab shows a view of the previously hidden ITAdminQueue table in the SQL database, which contains all sorts of back end commands that are queued up / in process to the BlackBerry handhelds.

The view above results from setting the software configuration of my test BlackBerry to a pre-configured RSA SecurID application, turning the wireless off, and manually pushing it out via the "Deploy Software" command. You can see there are two entries listed under this new tab, which don't show a whole lot of information. I assume because they are simply queued, and have not errored out yet.

Here are the corresponding ITAdminQueue rows in the actual SQL table:

Note that the first entry is actually listed with a Command column number of 129. This corresponds to the BES Policy service's SEND_APC_ITPOLICY command. The IT Policy is always sent first whenever an application is delivered, as it includes any software application policies you have created. As these policies need to be applied before the app is installed, sometimes there are errors applying the IT policy - which can prevent an app from being delivered successfully.

Note also that this IT Policy command does not show up in the Software Configuration Status tab, which leads me to believe this tab is built using a filter to only show software application entries, not the whole table.

Below this row there are two additional rows listed, both showing the Command column set to 128. 128 corresponds to the SEND_APC_APP command, so these are the commands to send the application itself. Note that these rows in the SQL table do not have a Time Sent value, but the IT Policy does. More proof that the app will not be sent until the IT Policy is applied successfully.

One interesting thing is that the app I was pushing contained only one .cod file, so why two SEND_APC_APP commands / rows? Looking into the Policy log we find something interesting:

Queuing the package RSA SecurID, consisting of 1 modules in 2 parts.

So I think this this one .cod module is split into 2 parts for transmission to the BlackBerry, for what reason I do not know. Perhaps it was too big to send all at once?

Although this new tab of information is helpful, I see it does not allow you to delete any queued commands. Some software push issues have been resolved by running the following SQL script:

osql -S [SQLServerName] -d [DBName] -U [SQLLogin] -P [SQLPassword] -Q "delete from ITAdminQueue where (command = 128 or command = 129) and status = 6"

This command will clear all SEND_APC_ITPOLICY and SEND_APC_APP entries with a status of 6, which designates an error; this allows future pushes to proceed properly.

Some other BES Admins have recommended just deleting all rows referencing a particular PIN number to resolve issues with an individual BlackBerry not receiving an application properly. I cannot vouch for this method however, as I have not used it. I do occasionally run the SQL script above during standard maintenance of our BES environment, or if a particular application push problem occurs.

From RIM's documentation, here are some error codes you might see in this new tab:

* Device timed out waiting for module
* Device reported insufficient memory to install module
* Device reported insufficient privileges to install module
* Device reported Invalid Version in packet, supported version is %s
* Device reported that there was an Incomplete Module
* Device reported that the Module Save Failed
* Device reported a general failure installing the module
* Incomplete ACK data for APPD request
* Device reported a %s error while installing module
* Device Reported Data Format Error in packet while installing module
* Device Reported Invalid Command while installing module
* Device Reported Insufficient Body Data while installing module
* Device Reported Invalid Module Hash while installing module
* Device Reported Invalid App Data Length while installing module
* Device Reported Insufficient App Data while installing module

You can check out the support article, KB15470, for more information on these messages and there resolutions.

I am sure that this new tab will help in resolving future application push issues, however I am also sure that some manual cleaning of the ITAdminQueue table will still be needed occasionally as well!

Heading to WES 2008

2008-05-06T11:01:00.001-07:00

Well I know some of you who read this blog will be at WES this year, looking forward to meeting all of you in addition to the great bunch of BES Admin folks that convened last year.

Generally our meeting place was in the right rear of the solutions showcase but we may also be hanging out around the BoxTone lounge, check out the WES 2008 thread on www.blackberryforums.com BES Admin corner for the latest details...

Hoping to see you there!

Address Book Contact Labels and the BlackBerry

2008-05-05T11:57:00.000-07:00

Had an interesting issue crop up today with an end user who changed the contact labels associated with an address book entry. Here is the default when you click the Edit Contact Labels button within your address book entry form:

This person had modified the labels to reflect different information, for example changing the Pager label to Home Phone #2. The problem is that the label change does not sync over to the BlackBerry, which shows the Label still listed under Pager.

I called RIM and verified this is the way it should work, as the underlying field name does not change, and RIM maps to the field name, and not the changeable display name.

I understand the issue and agree with RIM's assessment, however this is one of those gray areas where it is difficult to determine who is responsible for the issue. Is it Lotus for providing the (perhaps unwise) ability to change display names to something that is completely different from what the underlying field name reflects? Or is it RIM for not supporting this Lotus Notes feature, even though it may not be the best of features to have from a data integrity standpoint.

As is generally the case, it is perhaps a little of both, which is why these things tend not to get fixed as each side points to the other and neither will take the responsibility to address it.

This is not that big of a deal to me, and personally I agree with RIM's assessment that their sync engine should map to the field name and not a nebulous display name that can be changed at will by the end user.

However, this is just a small example of the types of conflicts that can happen when you have two separate software entities interacting with each other, creating a no-man's land of responsibility with the customer trapped in the middle.

IT Policy Removal Options

2008-03-28T07:04:00.000-07:00

For security reasons, the IT Policy assigned to a device cannot be removed, even if the device is wiped. It is like a permanent tattoo. Like tattoos, however, they can be removed with a long and painful procedure. At least that is the way it used to be, nowadays we have more options. Let's go over them.

1) Wipe the device. Although the IT policy still exists on the device, the wipe process opens a "window" where an activation against another corporate BES server can overwrite the IT Policy with the new one. So for moves between organizations, you can only replace one policy with another, you cannot overwrite. For people who purchase previously corporate BlackBerry's on EBay for personal use this has been quite a hassle, as they are stuck with the old corporate policy which may be locked down significantly.

Which leads to...

2) Use the policy.bin file to clear the policy. This is a long and messy procedure (which I will not get into here) which mostly gets rid of the old policy, using the pre 4.0 method of a policy.bin file and Desktop Manager. I don't like this method because of the time it takes, and the lack of being in a known consistent state at the end. It is not really "just like new" when this is done, but can be enough for most people.

3) This is a new one. With a device OS later than 4.2.2, you can use the "Remote Wipe Reset to Factory Defaults" procedure documented in an older post here:

http://besdomino.blogspot.com/2007/10/remove-it-policy-new-feature-of-41-sp4.html

4) This is also a new method, which does not require an IT Policy assignment or a BES server at all, just the Javaloader utility in the RIM developer tools download. Note that an OS of 4.3.0 or higher is required, and the Javaloader utility itself must be the latest version.

Strangely, the utility does not list version numbers, so you just have to look for the available option:

resettofactory
Reset IT policy to factory settings

...when you run the executable without parameters to see if it supports it. If it does, then this command will wipe the IT Policy right off, and set it back to default:

c:\javaloader -u resettofactory

The future of IT policy removal is a bright one, indeed.

Multiple BlackBerrys per Mailfile?

2008-03-21T12:55:00.000-07:00

Answer: Yes with some caveats.

First, you need to have fully separate BES environments, i.e. at least 2 SQL back end databases. This is easy for us as we have an EU BES environment on it's own SQL database.

If we need to set up an additional handheld for an end user, whether they need it temporarily for an assistant, or need to test a new device, then we add their account to the other BES environment and activate their device as normal. Since BlackBerrys are international devices there is no problem getting back to the device here in the U.S to finish activation.

Notes:

1) BlackBerry activation can take considerably longer due to the WAN if you are using another continents environment like we are. Don't freak out if it takes a few hours.

2) If you are doing OTA activation, temporarily disable the original BES account as it might pick up the activation email and discard it, leaving your remote BES server waiting forever. [BES servers are selfish and will NOC-block you if you let them. :)]

3) If these requests becomes common, you might want to think about a secondary local BES environment, perhaps running with an integrated MSDE, to support this service. Of course you need an additional SRP to do this.

4) Remember that these accounts cannot be failed/moved over to other local BES servers in a DR scenario, as they are on a separate database.

5) Don't forget about these "ex-pat" accounts if on a foreign BES server. The local admins may not ever delete them b/c they don't know who they are. So you should keep track of them and delete them when it looks like the service was cancelled or otherwise abandoned.

Other than these caveats, I have not seen any issues with this in our environment.

Managing the BlackBerry Device Firewall Remotely... Possible?

2008-02-11T09:52:00.000-08:00

Over the past few weeks as I was playing with setting device options remotely, I made a few mental notes when encountering databases (i.e. configurations) backed up to the SQL db. One of these was 'Firewall Options'.

I particularly noted this one because I have been looking for a solution to the firewall popups that appear on our user's devices. Overall of course the firewall is a good thing, but I like to keep the experience of our users seamless whenever possible, and incomprehensible (to the user) firewall popups do not make for a seamless experience.

One example of this is the RSA SecurID software for our BlackBerry devices, which we deploy so our users do not need to carry around a hardware token. The first thing this software does when installed on the device is listen for a software token which is wirelessly pushed to the device via a separate app.

This listening process constitutes a 'server' on the device, so upon installation we see the following firewall popup:

Now if I am pushing this software from the server to a device out in the field, then based on this screen the end user will be confused as to what to do. 80% of the time they may choose correctly and scroll down to click allow, but that other 20% may result in a support call.

So is it possible to:

1) Set the 'Allow' rule in the device's 'Firewall Settings' database in SQL
2) Push the 'Firewall Settings' configuration down to the device
3) Push the application and have it immediately begin listening for a token without user input?

Let's find out!

First we will monitor the SYNC log again to see if anything gets backed up to the server when an option on this firewall prompt is selected.

The first thing I notice is that if I leave the "Don't ask this again for:" box unchecked, then nothing happens... I see no log entries on the server side. If I reset the app and enable the listener again, this time choosing to check the box, however:

... voila! I do see something coming through to the server:

[60000] (02/11 11:50:51.111):{0x1710} [ODBCRecord::DoSetValue] DATA = "DbVersion[0] Data[0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000] UserConfigID[17] DatabaseName[Firewall Options] UID[1] ".

This tells me that my hunch was correct, and that any saved settings are stored in the 'Firewall Options' configuration, and therefore backed up to the server. Good news!

So let's now reset everything and query the 'Firewall Options' configuration to see the default value as compared to the updated settings:

0x1100000005000004000003000002000001000000 (Default)
0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000 (RSA Allowed)

It looks like the first numbers were changed from 11 to 27, then a bunch of numbers were inserted, extending the configuration string. I am guessing that this new set of numbers uniquely identifies the SecurID server application and the 'Allow' setting I chose.

If we reset the app, then choose 'Deny' here is what the string becomes:

0x270000010107536563757249440001046874747000060000000005000004000003000002000001000000

Everything is the same except for a single number, the '2' about 2/3 of the way through the string has changed to a '6'. So perhaps a '2' means allow, and a '6' means deny. Let's try clearing everything out, and then pushing the 'allow' string to the device before the software push:

use BESConfig
UPDATE SyncBackupRestore SET Data=0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000 WHERE UserConfigID = 17 AND DatabaseName = 'Firewall Options'
UPDATE SyncConfig set SyncType=1 WHERE UserConfigId = 17 AND SyncDataSourceId=11109
go

Now we reload the application, and...

Bummer.

Guess I'll have to keep digging on this one...

[Update 2/13/08]
I have been gently reminded of the use of application control policies by a fellow blackberryforums.com member, ACJones. By setting a policy on the software configuration for the RSA SecurID software, and modifying the 'Internal Network Connections' option from 'Prompt' to 'Allow', you can avoid the initial firewall prompt completely upon OTA installation!

I have tested this and it works flawlessly. Thanks AC!

Managing BlackBerry Configuration Settings Remotely - Putting it all Together

2008-02-06T10:37:00.000-08:00

Now that we know how to update the device settings, let's see how it works in a real world scenario. I have just nuked, reloaded, and reactivated my test 8700g with 4.2 OS - here is the before shot of the relevant configuration screens:

Now let's push down some common changes to our device settings:

1. [Messages Options / General Options] Hide Filed Messages = Yes (default = No)
2. [Messages Options / General Options] Hide Sent Messages = Yes (default = No)
3. [Messages Options / General Options] Keep Messages = Forever (default = 30 Days)
4. [Messages Options / Email Reconciliation] Delete On = Mailbox & Handheld (default = Handheld)
5. [Calendar Options] Initial View = Week (default = Day)
6. [Calendar Options] Keep Appointments = Forever (default = 60 Days)
7. [Address Book Options] Sort By = Last Name (default = First Name)
8. [Options / Date/Time] Time Zone = Central Time (-6) (default = Casablanca (GMT))
9. [Options / Screen/Keyboard] Backlight Timeout = 2 Min. (default = 30 Sec.)

These modifications will require the following sequence of SQL updates (you will need to replace the '15' UserConfigID with your target account's UserConfigID in your SQL DB) :

use BESConfig
UPDATE SyncBackupRestore SET Data = 0x550000010101000001010101000301010800014465736B746F70000A000
2543336323538313330000100030101000A00020005FFFF01000601020008
01000100090102000B000002000C000002000D000001000E00000000
WHERE UserConfigID = 15 AND DatabaseName = 'Message List Options'
UPDATE SyncBackupRestore SET Data = 0x270000000000001C02FC0300000000010000000F0000000101000500000
00100FFFF0000000F00000001 WHERE UserConfigID = 15 AND DatabaseName = 'Calendar Options'
UPDATE SyncBackupRestore SET Data = 0x0A000001010100000000000100 WHERE UserConfigID = 15 AND DatabaseName = 'Address Book Options'
UPDATE SyncBackupRestore SET Data = 0x040001000000000400021400000001000302 WHERE UserConfigID = 15 AND DatabaseName = 'Options' AND UID = -447168820
UPDATE SyncBackupRestore SET Data = 0x1000019A71C67ACBC67AB400000002000000000E00018131D8CDC28BB8
B90000000700000D0001D30CB969B820BFDD0000000600100001C2C1986B
11CE32B000000002000000780D0001BDAAFA49FFAD48CC00000006011000
013E3CBC03163D39CF00000002000001C24F0001CB2088784AFC39CC0000
000000416A6176613A2F6E65742E72696D2E6465766963652E7468656D652
E626264696D656E73696F6E5F7A656E5F333230783234305F772E5468656D
65466163746F72790D0001416028234B233EA70000000600100001F5006E1
93847CE260000000200000001100001BD91E3D465E4816900000002000000
00100001EBB9DE74CE5F773800000002000000322600011F0091A81B93E1C
30000000000186E65745F72696D5F6170706C69636174696F6E5F6D656E75
08000138D4D5F7DC80C5472E0001DA36AA198C1233A40000000000206E65
745F72696D5F62625F70726F66696C65735F6170702E50726F66696C65734
F00019B05812B5A1A15DC0000000000416A6176613A2F6E65742E72696D2
E6465766963652E7468656D652E626264696D656E73696F6E5F7A656E5F33
3230783234305F772E5468656D65466163746F72790D0001F33AFBD074DB0
7F100000006001000016B506A6FCD9D596A00000002000000011000012ECB
C575BEF57F3E00000002000000C80D0001094AC736D16697C800000006001
000011762E2440C4D5D510000000200000064080001E8D6CB484814CA60
WHERE UserConfigID = 15 AND DatabaseName = 'Options' AND UID = -1321243196
UPDATE SyncConfig set SyncType=1 WHERE UserConfigId=15 AND SyncDataSourceId=11109
go

This will kick off a one way sync from the server to the device, pushing all the configuration databases managed by the Backup component. It will take less than a minute for the SYNC process to pick up on this change, and about another 30 seconds or so to re-sync all the settings. [sleep 120] Now back to our handheld... it appears that all changes (in green boxes) have been picked up:

Now we must be sure to run the following command to restore the device to server synchronization:

UPDATE SyncConfig set SyncType=3 WHERE UserConfigId=15 AND SyncDataSourceId=11109

If we didn't do this, then any configuration updates from the handheld would not be backed up to the server. Actually, this might be an interesting option to employ for some firms who want total control over the handheld... one could schedule a daily push to refresh settings to the organizational standard settings to discourage end-user customization.

All of these commands could be easily scripted by putting them in a text file such as 'PushSettings.sql' and using the OSQL command's -i switch, i.e.:

osql -S sqlservername.company.com -U sqlaccountname -P sqlpassword -i PushSettings.sql

So that's it... We've gone down the rabbit hole into the depths of the BES SQL DB in order to discover some of it's secrets, and we came away with a method of pushing normally hidden device configuration settings over the air. Have fun, but remember to backup your SQL database regularly and TEST TEST TEST your scripts outside of production first!

Managing BlackBerry Configuration Settings Remotely - Part Five

2008-02-04T09:59:00.000-08:00

Here are the options for the Calendar, first with default settings, then with the individual options enabled along with the resulting Data string:

[Default Settings Above]
0x270000FFFFFFFF1C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[Initial View: Week]
0x270000000000001C02FC030000000001000000
0F0000000101000500000001003C000000000F00000001

[Initial View: Month]
0x270000000000001C02FC030000000001000000
0F0000000201000500000001003C000000000F00000001

[Initial View: Agenda]
0x270000000000001C02FC030000000001000000
0F0000000301000500000001003C000000000F00000001

[Initial View: Last]
0x270000000000001C02FC030000000001000000
0F0000000401000500000001003C000000000F00000001

[Enable Quick Entry: No]
0x270000000000001C02FC030000000001000000
0F0000000000000500000001003C000000000F00000001

[Default Reminder: None]
0x270000000000001C02FC030000000001000000
FFFFFFFF0001000500000001003C000000000F00000001

[Default Reminder: 0 Min]
0x270000000000001C02FC030000000001000000
000000000001000500000001003C000000000F00000001

[Default Reminder: 1 Hour]
0x270000000000001C02FC030000000001000000
03C000000001000500000001003C000000000F00000001
(Minutes in hex: 3C = 60 decimal)

[Default Reminder: 1 Week]
0x270000000000001C02FC030000000001000000
060270000001000500000001003C000000000F00000001
(Minutes in transposed hex: 2760 = 10080 decimal ; 10080/ 60 = 168 hours / 24 = 7 days)

[Snooze: None]
0x270000000000001C02FC030000000001000000
0F000000000100FFFFFFFF01003C000000000F00000001

[Snooze: 1 Min]
0x270000000000001C02FC030000000001000000
0F0000000001000100000001003C000000000F00000001

[Snooze: 10 Min]
0x270000000000001C02FC030000000001000000
0F0000000001000A00000001003C000000000F00000001

[Snooze: 30 Min]
0x270000000000001C02FC030000000001000000
0F0000000001001E00000001003C000000000F00000001

[Start of Day: 8 AM]
0x27000000000000E001FC030000000001000000
0F0000000001000500000001003C000000000F00000001
(Minutes past midnight in transposed hex: 01E0 = 480 decimal ; 480/60 = 8 hours)

[End of Day: 6 PM]
0x270000000000001C0238040000000001000000
0F0000000001000500000001003C000000000F00000001
(Same as above, 0438 = 1080 decimal ; 1080/60 = 18 hours or 6 PM}

[First Day of Week: Mon]
0x270000010000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[First Day of Week: Tue]
0x270000020000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[Confirm Delete: No]
0x270000000000001C02FC030000000000000000
0F0000000001000500000001003C000000000F00000001

[Show Free Time in Agenda View: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000000010F00000001

[Show End Time in Agenda View: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000000

[Keep Appointments: 15 Days]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001000F000000000F00000001

[Keep Appointments: Forever]
0x270000000000001C02FC030000000001000000
0F000000000100050000000100FFFF0000000F00000001

[Wireless Synchronization: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000000003C000000000F00000001

[Show Tasks: Yes]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000100000F00000001

Now for the Address Book:

[Default Settings Above]
0x0A000001010000000000000100

[Sort By: Last Name]
0x0A000001010100000000000100

[Sort By: Company]
0x0A000001010200000000000100

[Confirm Delete: No]
0x0A000001000000000000000100

[Allow Duplicate Names: No]
0x0A000000010000000000000100

[Wireless Synchronization: No]
0x0A000001010000000000000000

And finally, Tasks:

[Default Settings Above]
0x0F000000000000000000000101FFFFFFFF01

[Sort By: Priority]
0x0F000001000000000000000101FFFFFFFF01

[Sort By: Due Date]
0x0F000002000000000000000101FFFFFFFF01

[Sort By: Status]
0x0F000003000000000000000101FFFFFFFF01

[Confirm Delete: No]
0x0F000000000000000000000001FFFFFFFF01

[Snooze: 1 Min]
0x0F0000000000000000000001010100000001

[Snooze: 10 Min]
0x0F0000000000000000000001010A00000001

[Snooze: 30 Min]
0x0F0000000000000000000001011E00000001

[Wireless Synchronization: No]
0x0F000000000000000000000101FFFFFFFF00

Managing BlackBerry Configuration Settings Remotely - Part Four

2008-01-31T12:03:00.000-08:00

Now that we know how to update the settings on the device, here is a screenshot showing the default settings, and a list of Data strings with all the different Message List 'General Settings' options:

Default Message List / General Options settings:

[Data String for Defaults above]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Time: NO]
0x350000000101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Name: NO]
0x350000010001000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Message Header On: 1 line]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B010002000C000002000D000001000E00000000

[Display Message Header On: 2 lines]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B020002000C000002000D000001000E00000000

[Display Message Count: None]
0x350000010101000001010001000301010200051E0001000600020008
00000100090102000B000002000C000002000D000001000E00000000

[Display New Message Indicator: No]
0x350000010101000001010001000301010200051E0001000600020008
01000100090002000B000002000C000002000D000001000E00000000

[Confirm Delete: No]
0x350000010101000001010000000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Hide Filed Messages: YES]
0x350000010101000001010101000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Hide Sent Messages: YES]
0x350000010101000001010001000301010200051E0001000601020008
01000100090102000B000002000C000002000D000001000E00000000

[Make PIN Messages Level 1: No]
0x350000010101000001010001000300010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Auto More: No]
0x350000010101000001010001000301000200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 15 Days]
0x350000010101000001010001000301010200050F0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 2 Months]
0x350000010101000001010001000301010200053C0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 3 Months]
0x350000010101000001010001000301010200055A0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 4 Months]
0x35000001010100000101000100030101020005780001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 5 Months]
0x35000001010100000101000100030101020005960001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 6 Months]
0x35000001010100000101000100030101020005B40001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: Forever]
0x35000001010100000101000100030101020005FFFF01000600020008
01000100090102000B000002000C000002000D000001000E00000000

[SMS and Email Inboxes: Combined]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C010002000D000001000E00000000

[SMS and Email Inboxes: Separate]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C020002000D000001000E00000000

The first line contains all the default settings from my test device which is an 8700g with the 4.2 OS software. The latter lines are the defaults with only the option shown enabled. The highlighted numbers in the rows show which number controls the setting in that row.

Keep in mind that this may be different from what you see depending upon your device type and OS version, however I suspect they would be similar if not the same for the same OS version.

Next: Calendar, Address Book, and Tasks options database settings.

Managing BlackBerry Configuration Settings Remotely - Part Three

2008-01-31T09:39:00.000-08:00

To begin, of course, a Disclaimer: The BES SQL database is the heart of your BES environment. Making changes to it outside of the UI is not supported by RIM and can do irreparable harm to your entire BES environment. The following is for informational purposes only, and I am not responsible for the thousands of midnight telephone calls you will get from your BlackBerry users when an errant SQL command has rendered your BES servers mere hunks of plastic and metal, and transmogrified into a CLM [career limiting move]. I for one will not be implementing these findings in my production environment! Now, onto to the madness...

Since we have discovered where the configuration setting "Hide Sent Messages" lives in the SQL database, let's try to modify it and get it pushed out to the device. First, let's reset the option back to the default of "No" and restart the wireless to get it changed in the SQL database. Here is the device screenshot showing the default settings:

And here is the SQL data, showing that the '1' has been changed back to a '0':

0x350000010101000001010001000300010200051E0001000600020008
01000100090102000B000002000C020002000D000001000E00000000

Now, let's use a SQL command to update this string to change the '0' back to a '1':

Perhaps if we restart the wireless on the device the change will sync down?

Alas and alack, the change did not come down. Oh well, that would have been too easy. Let's take a deeper look into the SQL tables to see where we can go from here. I see a promising table named SyncConfig...and it appears to... wait for it... hold the Synchronization Configuration settings! Just what we need:

I see there are many lines associated with UserConfigId 9, which is my single test user. I also see a column titled SyncDataSourceId, I wonder what the numbers in this column refer to?

After doing some digging I discover the SyncDataSource table, which contains mappings between the numbers in the SyncDataSourceId column and the names of these sources.

What we are interested in would be the Backup data source, or number 11109, as this is what we are working with. This leads us to row 102 in the SyncConfig table.

Now the question becomes, what do the other columns in this table mean, and what can we do with them? SyncType and ConflictResolution appear especially intriguing given what we are trying to accomplish here.

By changing some settings on the device and watching the effect on this table for other unrelated rows, I discovered that the SyncType field has three options available:

1 = Server to Device
2 = Device to Server
3 = Bi-directional

In our case, the 11109 Backup row in this table indicates '3', or Bi-Directional sync. I would have expected to see this set to '2' for one way backups from device to server only. However '3' may make sense, as the backup is from device to server for the most part, but can be from server to device when changing devices for instance.

Similar to SyncType, the ConflictResolution column appears to have two options available:

0 = Server Wins
1 = Device Wins

Our row 102 shows a '0' in this column, which again is a little unexpected as I assumed that the server side was a backup to what is on the device, so the device should always win in a conflict. Perhaps there are good reasons for these non-obvious settings that I am not aware of.

In any case, we have still updated the data in the SyncBackupRestore table but it has not sync'ed to the device, even though the sync settings shown above indicate that a) sync is bidirectional and b) the server will win on a conflict. So really this should just work without any changes, however we need to figure out some way of 'kicking' off a sync.

We know that when you disable and then re-enable synchronization from the device or BES manager for other functions such as Address Book, it will kick off an initial sync. The issue here is that there is no way through the GUI of the device (or the BES Manager) to kick off a Backup sync, as this is an underlying process not available to the end user or administrator.

Why don't we try to just flip the ServerEnabled setting from 1 to 0, and then back to 1 to see if this does it:

Unfortunately, looking at the logs it doesn't appear that this trick did anything. I see the following lines in both the SYNC and CBCK logs:

[46077] (01/31 13:09:36.021):{0x1178} [SYNC-Gate] Finished user config change check. No change detected.
[40039] (01/31 13:20:21.856):{0x1EE0} CheckResponseRcvd - no changes detected for any users.

Let's move on and try to change another field, how about we try to change the SyncType from '3' (bi-directional) to '1' (server to device):

Wow! Within a minute or so of making this change there was a flurry of activity in the SYNC and CBCK logs. I notice this line as well:

[46078] (01/31 13:26:53.092):{0x1178} [SYNC-Gate] Finished user config change check. Change detected.

Turns out this triggered a full synchronization, such that everything on both sides was compared and the changes pushed down to the device. Going back into our Message List Options screen, we are presented with:

As Dora would say, "Lo Hicimos! We did it!"

So in summary, we were able to push this change to the device by doing the following:

1) Locating the option in the data string of the particular settings database within SyncBackupRestore
2) Updating the data string with the particular option set, via SQL commands
3) Modifying the SyncType from '3' to '1'

Now, like a good boy scout, we must remember to clean up after ourselves. To get this to work we had to break the default SyncType, so let's set it back to '3' so that any future changes on the device continue to be backed up:

Next Up: Decoding the Data string in the "Message List Options" database to map out all the different option settings.

Managing BlackBerry Configuration Settings Remotely - Part Two

2008-01-30T12:58:00.001-08:00

OK so let's dive in and see what we can do with the SQL table SyncBackupRestore. First of all it looks like there are separate entries for different types of settings. Let's start with the entry for Message List Options for UserConfigId #9, which is the only user in this test environment:

It looks like the Data column is encoded in some sort of binary format, let's see if we can use OSQL to see what's in there:

So after a bunch of header information we finally see the data string, and it looks like this:

0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

As it exists now, this string represents the state of the options for the Message List, i.e. the Messages screen on the BlackBerry. The best way to discover what pieces control which device settings is to change an option on the device, wait for it to get synced up to the server, then check this string again to see what is different.

We will first test by changing a single option on the BlackBerry, Hide Sent Messages from the default of No to Yes (a common enough request!). Now the issue is waiting for the device's backup agent to push the change to the BES server, however I discover to my pleasant surprise that turning off the antenna and turning it back on will trigger the handheld agent to push any device changes up to the server. (Restarting the BlackBerry has the same effect).

I have been using the Unix "tail" utility to follow the BES server's CBCK log file (Backup connector), which is set to the debug logging level of 7 through the registry (4 is the max via the UI). When the device connects to the network we see the following update come through:

[60000] (01/30 14:26:07.086):{0x151C} [ODBCRecord::DoSetValue] DATA = "DbVersion[1] Data(first 32 bytes)[0x350000010101000001010001000300010200051E000100060102000801000100] UserConfigID[9] DatabaseName[Message List Options] UID[1] ".

This log entry only shows the first 32 bytes of the updated field, so we have to go back to the OSQL utility to display the full field again, and it is now set to the following:

0x350000010101000001010001000301010200051E0001000601020008
01000100090102000B000002000C000002000D000001000E00000000

The only change in this string is the highlighted '1' near the end of the first line has now been changed from a '0' previously. This leads us to believe that this number controls the Hide Sent Messages option on the device, and that modifying it from '0' to '1' enables it.

In the next post we will try to update this field in the SQL database and (somehow!) get that update pushed to the handheld.

Managing BlackBerry Configuration Settings Remotely - Part One

2008-01-30T12:28:00.000-08:00

Much of the configuration of BlackBerry devices can be controlled through IT Policy options, especially those having to do with security. However there are many more configuration options available on the device that don't have IT Policy correlates. In these cases it can be difficult to control the user experience or to enforce standard organizational settings.

There are a few options to automate the configuration of these device settings:

1) Configure a BlackBerry with your custom settings, then back it up to a .IPD file. For each new BlackBerry deployed, restore the .IPD file before activation.

2) Configure a BlackBerry with your custom settings, then assign it to the new user's account. Then activate a new BlackBerry against the same account, which will inherit the device configuration settings from the old device.

These options can set an initial standard, however they cannot be changed remotely or locked down and enforced over time. The BlackBerry device user will have the ability to modify these settings and break the standard.

The fact that option #2 above exists tells us that the device configuration settings are stored within the BES environment, as that is how swapping in brand new device for a user can inherit all the old device's settings. Unfortunately the location of these settings is not easily accessible to the BES Administrator, as this function was really only designed for backup and restoration purposes.

However... if we can figure out where these settings are stored perhaps we can modify them somehow from the back end, and push them down to the device. We know that most configuration information is stored in the central SQL database, so this is where the configuration must be stored as well.

Peeking around my test environment I discovered a table in the database called SyncBackupRestore. It appears that much of the device's configuration settings are stored in this table. In the next post we will dig deeper into this table to discover what it holds and how perhaps we can modify it and push the changes to a remote BlackBerry.

BlackBerry Lookup - The Details (Updated for 4.1 SP4)

2008-01-23T10:40:00.001-08:00

After doing some troubleshooting on lookups the other day I discovered that there are now some additional fields that are returned via lookup, that were not there back in the 4.0 days when I posted my first post on this over a year ago.

Specifically, the Home Address fields have now been added, which is nice. Here is a lookup result that includes as the data the Notes Person document field names that the lookup pulls from for reference:

Please note two missing items from this lookup:

1) PIN is missing due to the test Person document not having a BlackBerry device

2) The "Fax: OfficeFAXPhoneNumber" lookup result is missing, though populated in the Person document. This is due to a known issue introduced in SP4 MR1 and scheduled to be fixed in SP5.

Now, onto something interesting that I wasn't aware of before. You are able to customize the lookup results to a certain extent, by using the 4 "User Defined x" fields present in the "Edit PIM Sync Global Field Mappings" command from the BES Manager. [I didn't think this was an option as I always thought that the Global Field Mappings had to do with Address Book synchronization, I never thought they would impact the lookup results.]

Simply choose an unmapped field, drop down the Device Field column field and choose one of the User Defined fields to assign it to:

In my test environment I have assigned the following custom fields:

User Defined 1 --> Assistant
User Defined 1 --> Manager
User Defined 1 --> Spouse
User Defined 1 --> Children

When I do this, I get the following lookup result:

I see the Assistant, Manager, and Spouse fields, but not the Children field I mapped. Perhaps there is an issue with the "User Defined 4" field not working correctly for lookups. Let me change the mapping of this field to "Department" and see what happens:

Interestingly, the Department field does come over. So perhaps it is not an issue with the "User Defined 4" field , but with the Children field not working correctly.

On a final note, the Comments field on the Person document can be multiple lines and all lines will show up on the BlackBerry lookup. So instead of the "User1:" etc mapping above you can create freeform text lines in the Comments field if you need them on the BlackBerry lookup. For example, I added two additional lines to the Comments field in the Person document, and here they are on the lookup:

I'm not sure what the limit on the number of lines or characters are for the free form text Comments field. I was able to put 60 lines of information in that field and they were all returned to the BlackBerry on a lookup, so there is quite a bit of information that could be stored here.

Well I hope this helps you customize your lookups so that you can give your users better information directly on their device. Now the greater problem is how to get all this good information from other sources, such as the HR database, into the Person documents in names.nsf in the first place. (Luckily for me that heavy lifting job was already done by our development team for another project.) Good luck!

Lookup Strangeness -> "Reset PIM Sync Global Field Mapping" is the Fix

2008-01-22T10:19:00.000-08:00

I recently discovered that our BlackBerry lookups were returning some bad information. Our addressing information was coming up wrong, in that the Home Address header was missing, replaced by a second Work Address header. Also, the information in the home address fields was limited just to city and country.

Here is what we were seeing (fields filled with dummy info for testing):

The last address is actually the two home address fields city and country, but labeled as a second Work Address. I had never seen this before, at least this was working properly a few months ago. We have a secondary BES infrastructure in the EU, totally separate, but alas this was occuring there too, which pointed to a Domino problem.

I assumed this must be due to some corruption or weird customization in the names.nsf, where the BES pulls this information from. However copying and pasting my person document into my test BES environment, then doing a lookup from a test device, returned all the proper information! Puzzling...

It is difficult to explain this issue to RIM support, and they also could not recreate the solution in their environment. On a whim, I decided to investigate the "Reset PIM Sync Global Field Mapping" command. After testing this in the test environment and determining that this command would not reset user's custom address book mappings, I ran it. Guess what? All lookup problems solved:

The Home Address is now properly labeled, and although blanked out above, the fields for street address, state, and zip code now appear along with city and country.

This solution points to the address book mappings being off kilter within the BES SQL database, but how did that happen? We don't touch them. And since both of our completely separate BES environments experienced the issue, I must believe that a particular upgrade path (we keep versions the same in US and EU) caused this mis-mapping.

Easy solution to a strange issue!

p.s. I discovered some other interesting lookup related information during this search that I will discuss in an upcoming post.

How to Recover* from a Total Loss of BESMgmt SQL database (* with caveats...)

2008-01-16T09:19:00.000-08:00

The BESMgmt SQL database is the heart of the Blackberry Enterprise environment. Without it, BES servers will continue to deliver mail but no meaningful changes can be made, or BES services restarted, until it is back up.

The only piece of the BES environment that is out of my control is the database, which is hosted and managed by our SQL team. Although I have full confidence in their replication, backup, and recovery plans, I still need to plan for the (hopefully very remote) possibility that the SQL db might go bye-bye someday.

Until now, this worst case scenario resulted in an "oh well, everybody will have to reactivate their devices" DR plan. I dread the day I have to say this to the CIO!

Recently, however I got wind of the idea of using the nbesmigration.exe utility for DR purposes. This utility was originally designed to populate user accounts in the 4.0 SQL database from a version 2.2 profiles database. Since the 4.x environment still maintains a BlackBerryProfiles notes database on each server, the tool could theoretically be used to re-create the user accounts from these profile entries into a brand new SQL database.

While running the tool in testing, I noticed the following text as part of the built-in help:

BlackBerry Enterprise Server(c) for Lotus Domino(c) Migration Tool, Version 4.1
Copyright (c) Research In Motion, Ltd. 2004, 2006. All rights reserved.
Modification date: May 8 2007

Usage: NBESMigration.exe [options]

This tool can be used to migrate entries from BESD 2.2.x Profiles database to BESMgmt database (UserConfig table). For migration, create the database first using CreateDB, create a server entry in ServerConfig, and run this tool from your BESD server's Domino directory.

This tool can also used for disaster recovery or fixup purposes on already installed BESD server (4.0 and up). Shut down the BES, and run this tool from your BESD server's Domino directory. The tool will check all user docs in Profiles DB, create entries, if needed, in BESMgmt DB (UserConfig table), and it will link entries using SQLID field.

From the text it appears that this tool does support a limited form of DR. Good news! So to test, I did the following in my test environment:

1. Detach the BES SQL db and drop all connections.

2. Stop BES server task and services.

3. Run the BlackBerry Congfiguration Utility and choose the Change Database option.

4. Enter a different database name, thus creating a brand new database.

5. Re-enter CAL and SRP information, but do not restart services.

6. From the tools subdirectory of the BES 4.1 installation share, copy the nbesmigration.exe executable to the c:\lotus\domino directory

7. Run the following command nbesmigration -d: "[SQL Server name]" "[SQL DB name]" -u: [SQL login] [password]. In my case this command line looked like this:

nbesmigration -d: "SQLTEST01" "BESMgmtTest" -u: besdbowner besdbpassword

The tool indicated that it found my one test account and migrated it successfully.

8. Restart the BES server task and services.

The server fired up and I was able to immediately send and receive messages, and do lookups from my device! So this is a great way to recover basic functionality in the case of a total loss of the SQL db.

Keep in mind that you need some information documented and handy to provide to the configuration wizard:

CAL licenses
SRP ID's
SRP Authorization keys

Now, for the caveats:

IT Policies are reset to defaults (!)
PIM Sync Mappings are reset to defaults
Email Settings are reset to defaults
Probably a bunch of other stuff is gone or reset to defaults

Although some of the other stuff is fixable, I was unable to successfully apply the default IT policy or a newly created IT policy in the new enviroment, even when it was named the same as the old IT policy. It always came back with an error. On the other hand, when I switched DB's back to the original, the original IT policy was able to be re-applied successfully.

The upshot is that this is a last-last-resort option to get mail flowing to the devices without having everyone reactivate. If you can later restore the original SQL DB and switch back to it, then IT Policies, PIM Sync settings, etc, can be restored back to their original good state.

If you can never get the original DB back, however, then you might have to move forward and find some creative solutions to the problems above. If the solution does eventually require handheld reactivation, well then at least you can control and stage it on your terms, without having hundreds of people screaming at you to get their BlackBerry email fixed!

Last Handheld Contact Times - Bogus?

2008-01-15T08:01:00.000-08:00

This is a useful field to know when the last time the server talked to the handheld... until I started to suspect this information, as there were contact times very recently with handhelds that I knew were disabled.

To test this out I turned off my BlackBerry and restarted the BES task. The last handheld contact time updated to the current time, with the device turned off in front of me! See the below screenshot, showing last contact time of 1/15/2008 10:16:51AM, when the device was turned off at about 10:10 AM.

To see why this is updating, I checked the MAGT logs for activity on my account during this time:

[40000] (01/15 10:16:51.733):{0xB04} {Test User/Domain} Sending PIM Transaction to Sync Server, Tag 225 (config request)
[40000] (01/15 10:16:51.733):{0x1F9C} [BIPP] Send data, Tag=225
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received status DELIVERED, Tag=225
[40392] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SRP: TID=225, type PIMSYNC returned DELIVERED
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received datagram, Tag=436
[40076] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SendStatusToWirelessNetworkUsingSRP SendQ, TID=436
[40400] (01/15 10:16:51.748):{0xB04} {Test User/Domain} Received datagram with content type agent_sync, TID=436, for user Test User/Domain
[40000] (01/15 10:16:51.748):{0x1F9C} [BIPP] Send status DATA_ACCEPTED, Tag=436

The only activity that occured was an internal communication to the Sync server, and what looks like a request to perform a handheld agent sync. Note that both transactions appear to show successful connections with the device, however these are only internal communications between components on the BES server itself. Perhaps the time of successful internal communication is being reported?

In any case, it appears that this is quite a bogus field which provides no useful real world information.

MDS What?

2007-11-02T12:59:00.000-07:00

MDS stands for Mobile Data Service, and it is a service that allows you to access other data sources (mostly Internet / Intranet browsing) via your BlackBerry.

In BES 4.1, however, MDS has turned into a many-headed hydra. What we know of as "BlackBerry Mobile Data Service" in 4.0 is now inexplicably called the "BlackBerry MDS Connection Service" in 4.1. Meanwhile, there are a whole bunch of new MDS related services:

•BlackBerry MDS Application Integration Service
•BlackBerry MDS Data Optimization Service
•BlackBerry MDS Provisioning Service
•BlackBerry MDS Administrative and Management Service
•BlackBerry® MDS Studio Application Repository

But wait, there's more! The new MDS stuff also requires a brand new SQL db running on the same SQL server as the BESMgmt database, which can results in permission issues when creating without the proper authority.

All of the above services & database are related to the new 4.1 software deployment environment installed when you choose the upgrade option called:

"BlackBerry Enterprise Server with MDS Services and Components"

But really, this whole new "MDS" software deployment infrastructure is not needed at the initial upgrade from 4.0 to 4.1. To avoid complexity, you can leave this whole new set of services out of the upgrade. Later, you can install the new MDS stuff on a separate server if you like, as it was made to be modular and have one separate MDS instance serve many BES servers.

The confusing part is that if you run the upgrade, you get these two choices:

If I do not know *exactly* what this means, I will by default choose the second option, "BlackBerry Enterprise Server with MDS Services and Components", because I want to keep my MDS service from 4.0, right?

Wrong... choosing the first option will give you the same 4.0 MDS (renamed MDS Connection Service) while avoiding the complexity of installing the new whiz-bang MDS software deployment stuff.

In a nutshell: if you want to greatly simplify your 4.0 -> 4.1 upgrade, opt for the first install method selected in the picture above. You will lose no MDS functionality from the 4.0 perspective, and can add on the new stuff later when you are ready and comfortable with 4.1.

Malformed Message Crashes BES 4.1 SP4

2007-11-01T07:09:00.000-07:00

The idea that a malformed message will crash a BES server is nothing new - service packs have taken care of these issues many times in the past. I apparently discovered another one, as one of my servers crashed twice just after midnight last night.

Fortunately Domino restarted itself and was back up and operational in minutes (thanks transaction logging!). After the second crash, however, it did not crash again. Usually the BES will keep trying to re-read the malformed message and crash over and over until you figure out the message and delete it from the user's mailfile, but not this time.

From the logs I see the attempts to read which resulted in crashes:

[40000] (11/01 00:00:44.858):{0x19B0} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:00:44.905):{0x19B0} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

But on the third attempt I see this:

[40000] (11/01 00:08:10.515):{0x1640} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM
[20039] (11/01 00:08:10.530):{0x1640} {User} Already attempted to open NID=3DAF2 for user User: Message has been quarantined, skipping now

Nice job RIM! This quarantining feature allowed me to stay peacefully asleep instead of having to get up and hunt down the offending message.

BTW, the message in question was a digest from a mailing list which included a BinHex encoded MIME part in the body of the message:

--B_3276667131_13573
Content-type: application/mac-binhex40; name="[Filename].doc"
Content-disposition: attachment;
filename="[Filename].doc"

(This file must be converted with BinHex 4.0)
:(8K[G#p1Eh3J9A"NBA4P)#dJ5R9XH5!R-$FZC'pM!&Fi3Nj08eG%!*!%UJ#3"GN
Jd-m4i+'a'Z%!N"!q!!-!r[m*!!B!N!X"!*!$8!#3#"!!!&)!N!-"!*!$r[q3!`#
3"%m!N!2rN2rrN,(XTF%!Kf%*"!!!q"+r!*!&!4%!!3!"!!B!!28L!!!1!'TLDQ+
`Zl#l!*!5#33@!#3d!!$Df3%!fYN"!28F!*!Hrrm2!*!*rrm2!*!*rrm2!*!4L!#

I am pretty sure this is the part that utterly confused the BES since:

a) it was in the body

and

b) was a binhex part, which I have seen trouble with in prior BES versions even when it was properly encoded.

Looking through the release notes for 4.1 SP4 MR2, I see the following:

*SDR 135729 In BlackBerry Enterprise Server Version 4.1 SP4, if a message contained truncated or incorrectly encoded data, the BlackBerry Enterprise Server might have stopped responding. In BlackBerry Enterprise Server Version 4.1 SP4 MR2 and later, this issue is resolved.

I love to see this - it means I don't have to report the issue to RIM, someone else already has! Also I can tell my manager when he asks about the crashes that it is already fixed in the next maintenance release, which we will now plan on deploying.

Decoding the BlackBerry State Database

2007-10-24T10:18:00.001-07:00

So we all know that state database correlates the messages in the mailfile with the messages on the BlackBerry. The way it does this is by creating a separate entry in the state database for each email the BES sees in the mailfile. Each of these entries has exactly the same UNID as the original email in the state database.

Now I am trying to decode the MessageState field in the state database entries. Searching through my own state database, I find all of the following various "state codes" under the MessageState field:

0
1
2
3
4
5
7
8
9
10
13
14
17

That's alot of "states" a message can be in!

I have started testing and have determined the first couple:

2: Email has been queued / sent to wireless network but not received by device yet
3: Email not redirected to handheld (redirection disabled)
4: Email has been delivered to device

It will be tough to figure out the rest, though! I will update this doc as I discover them...

Remove IT Policy - New Feature of 4.1 SP4 / OS 4.2.2

2007-10-08T10:34:00.000-07:00

We now have an easy way to remove the applied IT Policy from a handheld, which was previously locked to the device even after you wiped it.

If you have 4.1 SP4 and a device with OS 4.2.2 or greater, here is how to remove the IT policy and set the device to a true factory default state:

1) Create a new IT policy (or modify an existing policy) and set the Remote Wipe Reset to Factory Defaults setting in the Security Policy group to True

2) Assign this IT policy to a particular user account
3) Send the command "Erase Data and Disable Handheld"

4) Verify receipt of command by device under "IT Policy Status"

Update on Comcast's Man in the Middle Attacks on Lotus Notes

2007-09-27T06:14:00.001-07:00

For those of you Notes client users suffering from Comcast's "filtering", there is a great post by my colleague Kevin Kanarski who has details of packet captures that result from sending an email with a 6MB attachment from a Notes client using the Internet connection.

These captures, from both ends, clearly show that Comcast is imitating both the client and server in sending RST [reset] packets to the other end of the connection. Neither the client nor the server generated any RST packets, so this is definitely shady behavior by Comcast.

Tuning the "More" Cache

2007-09-20T12:04:00.000-07:00

We all know the pesky little "More" command that lets you download successively more and more of your email, beyond the first 4K delivered to the BlackBerry.

Well on the server side, there is a cache that holds this information, and it is called the "More cache". [Wow these names really need a little obfuscation, they make too much sense to exist in the PC world]

Anyhoo, you can tune the amount of cache that the BES allocates to this function, and by default it gives it 10MB. Well at this level I was filling up the cache in about 6 hours from server startup, and only realizing a maximum of 84% hit rate:

[45009] (09/14 02:35:58.078):{0x13BC} More cache hit rate: 0.0%, requests: 0, adds: 0, size: 0.0/10.0 Mb
...
[45009] (09/14 08:33:32.778):{0x13BC} More cache hit rate: 69.0%, requests: 126, adds: 2270, size: 5.7/10.0 Mb
...
[45009] (09/14 09:48:32.294):{0x13BC} More cache hit rate: 75.7%, requests: 214, adds: 5031, size: 10.0/10.0 Mb
...
[45009] (09/14 23:48:32.111):{0x13BC} More cache hit rate: 84.4%, requests: 784, adds: 29120, size: 10.0/10.0 Mb

If the remaining text is not found in the More cache, then the BES server needs to open the mailfile to pull it in, which is expensive in terms of network I/O and Domino server I/O. [Not hugely, of course, but I like to squeeze as much performance out of my systems as possible] If it can find it in the cache, here is what you will see in your MAGT log:

[40231] (09/14 01:11:01.290):{0x608} {XXXX} Original message (RID=-576019096) retrieved from More cache. Not opening the mail file.

That is a message that I like to see!

So I decided to double the default of 10MB to 20MB by adding the following DWORD registry key, and setting it to "20" decimal:

HKLM\Software\Research In Motion\BlackBerry Enterprise Server\Agents\MoreCacheSize

After a restart of the server (not required) here is what I now see:

[45009] (09/20 13:50:51.888):{0x14E0} More cache hit rate: 89.5%, requests: 3004, adds: 127078, size: 20.0/20.0 Mb

So fully loaded, it is running ~90% cache hit rate now, which is not a huge gain but it is something. RIM considers >90% hit rates to be good, but I haven't decided how much more I want to push this setting as it takes away memory from other processes for this task.

BlackBerry Service Books - BES only

2007-08-15T13:34:00.000-07:00

Here is a comprehensive list of the BES related service books available on a BlackBerry device:

Desktop [CMIME]: Send/Receive Email and Wireless Reconciliation
Desktop [SYNC]: Wireless Address Book
Desktop [CICAL]: Wireless Calendar
Desktop [ALP]: Address Lookup
Desktop [IPPP]: MDS Info for web browsing
Desktop [BrowserConfig]: MDS Info for web browser
Desktop [OTASL]: Over the Air Software Loading (4.2 device OS or later)
Provisioning [Provisioning]: Enterprise Activation

Note:

If you got something broke, then you may be able to quickly resolve it by deleting / undeleting a service book from the Blackberry device itself. This process regenerates / reconnects with the BES server in a magical way, requiring no server side intervention. Here are two common issues you can maybe resolve this way:

Wireless reconciliation such as read/unread marks, deletions not working, even though it says enabled from both server and device: Kill and Resurrect the Desktop [CMIME]

Address Book is not syncing, even though it says it is enabled from both server and device: Kill and Resurrect the Desktop [SYNC]

Comcast Blocking Lotus Notes Attachments over 2MB

2007-08-10T06:49:00.000-07:00

This is not related to Domino BES in any way, but thought I would share. It appears that Comcast (at least in the Chicago area) has recently installed a blocking system in order to combat P2P traffic, which has the inadvertent effect of blocking Lotus Notes attachment uploads over 2MB.

Using Lotus Notes with the internet passthrough, via a home Comcast cable IP connection, and trying to send any message over 2MB will result in a bunch of IP reset (RSET) commands sent by the Comcast blocking system which blocks the message from being sent.

We have received 2 calls in the last week regarding this issue, and 4 other tech people with Comcast cable modems have confirmed that sending attachments over 2MB no longer works. The error received is the following:

"Remote system no longer responding: [Server name] mail.box"

If you try to send a message just under 2MB you might see the following:

"Mail was successfully submitted for delivery but a copy has not yet been saved in your mail file due to server not responding"

Has anyone else received complaints from home / laptop Notes users about this issue?

UPDATE: This appears to be confirmed by a posting at www.dslreports.com which discusses Comcast's use of a filtering product called "Sandvine":

http://www.dslreports.com/forum/r18323368-Comcast-is-using-Sandvine-to-manage-P2P-Connections

BlackBerry Date/Time Source Explained

2007-08-02T12:51:00.000-07:00

You have three options of how to get the Date/Time synchronized automatically on the BlackBerry:

1. Set it to BlackBerry (the default), which gathers the information from the BlackBerry network, i.e. the RIM NOC.

There is some confusion about this, let's clear it up right now. Setting this to BlackBerry does not mean you have to set the date/time yourself on the BlackBerry. It does not mean that it gets the time from Desktop Manager when you cradle / sync. (that is so 2005 anyways, it's all wireless now baby!) It means that you get the date/time directly from RIM over the air, that's it. Here is the debug log lines from the device when you click Update Time:

guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:08 2007 severity:5 type:2 app:net.rim.timesync data:SynR
guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:08 2007 severity:0 type:2 app:net.rim.timesync data:Send
guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:10 2007 severity:0 type:2 app:net.rim.timesync data:Recv

2. Set it to Network, which gathers the information from the wireless carrier network, whether AT&T, T-Mobile, Verizon, Rogers, O2, Vodafone, etc. Here is the debug log lines from the device when you do this:

guid:0x1295B4AADE149AFC time: Thu Aug 02 15:15:55 2007 severity:5 type:2 app:net.rim.timesync data:SynR

Interestingly, when you click Update Time with the Network setting, it just copies the date/time recorded under "Network Time" and "Network Date", instead of directly querying the carrier network. I assume this "Network Time" is regularly synced through some background mobile radio process.

3. Set if to Off, which does not gather any info. You set it and maintain it yourself.

Please note that, as I explained in my prior post, none of these settings will update the time zone you are in to the local time zone when you travel. None of them. Stop trying to change from Network to BlackBerry and back and clicking "Update Time" over and over again. It just won't do it.

Automatic TimeZone Switching for BlackBerry

2007-08-02T12:18:00.000-07:00

The Motorola Q does it, the Treo does it, the Razr does it... and yet NO BlackBerry does it. What am I talking about? The seemingly obvious and basic cellular phone function of automatically switching the time zone for you while you travel.

This does not require internal GPS, as the other phones do not have/need it and yet have gotten their location / time zone directly from the network they attach to for years.

Now I have seen multiple discussions on the forums complaining/questioning about this feature, where "switchers" to BlackBerry just can't understand how to enable this feature on their BlackBerry... it is such an obvious feature that they think they are stupid and don't know how to get it to work, rather than the ridiculous notion that maybe this feature does not exist on the BlackBerry!

Arguments about setting the Date/Time source to Network vs. BlackBerry, the use of GPS, GPS not required, A-GPS vs internal GPS, it is just an endless circular conversation I have seen over and over with little resolution.

There have been justifications floated around that since the BlackBerry is a multifunction device, not just a phone, that the calendar items would be off whenever you traveled to a new time zone, causing mass confusion. For this reason they left it to the user to manually change the time zone.

This makes little sense to me, in that it might cause me mass confusion to have my calendar appointment times off from the local time I am in! In any case, RIM could even leave it to the user by default, but at least put it there as an option to turn on.

My theory on this, using no background information at all, is that RIM simply does not trust the wireless carrier networks to provide an accurate time zone setting from their networks. If one of the carriers passes out bad time zone data and messes up someone's entire calendar then RIM would be blamed. That is just a theory on my part, but that fear does not necessitate leaving out even the *option* of enabling this feature on the device.

So at WES a few months back I came armed with a couple of questions I was going to get answers to... one of them being this issue. The people I talked to - who granted were not the go to handheld development people - did not express any sort of justification for leaving the functionality off. In fact, they expressed the idea that "Yeah, that makes sense, we should look into enabling a feature like that."

There must be a back story I am missing here, there must have been some conversation about this at some conference table at some time in the 5 years since the first BlackBerry phone was released. Can someone enlighten me... please?

Dedicated Attachment Service Gotcha

2007-07-23T08:26:00.000-07:00

Early on we decided to separate our attachment service onto it's own dedicated server in order to offload that work from the BES servers. So now all 3 production BES servers point to this one attachment server.

One thing I noticed early on is that the BES servers cannot re-attach to the attachment server automatically. That is, if you restart the attachment server/service, attachment decoding will fail for all BlackBerry users until the associated BES task is restarted on all the associated BES servers.

The connection to the remote attachment server is handled within the BES Domino task, and the first time it connects upon startup, the BES task creates a special ID associated with that server's connection. When the attachment server is restarted, that connection ID no longer exists and the attachment server will reject decode requests from the BES server.

It is a simple fix (restarting BES task), but disruptive and can cause headaches during our monthly maintenance window - I have to always remember to restart the attachment server first before any of the BES servers are restarted.

This issue occurred in the 4.0 environment however it would be interesting to see if it still occurs in the 4.1 environment. When I find out I will post an update, however I haven't seen any SDR's addressing this issue as of yet.

One more thing to watch out for in a complex distributed BES environment!

UPDATE 7/24/07: It appears that this issue was addressed in 4.1 SP4, so no more reboot juggling required!

Thinking of removing your DST software config? Read this...

2007-06-20T06:14:00.000-07:00

Thinking that this was no longer a large issue, I nonchalantly removed the DST software config from ~1300 users last week. About a few hours later I get some inquiries from the regional offices to the effect of:

"We had some people call about receiving a 'Permission's restrictions need to be updated, do you want to restart your device now or later' prompt on their handheld. They accepted it and the device restarted, but I had never seen this before, what is it?"

This is the "Oh s***" moment, when you realize that something seemingly innocuous had directly impacted each and every user you modified in bulk.

You could say this was pretty minor, their functionality was fine and all they needed to do was restart the device, but that type of stuff doesn't fly in our environment. Everything that directly impacts the user requires a change control approval process.

Time to move the function "unapply software configuration" from daily administrative task to change-controlled lockdown!

Apparently this happens when you initially followed RIM's recommendation and created a specific software policy for DST which allowed external communications. Removing this software policy reverts it back to some default policy which requires a restart, hence the user prompt.

Oh well, live and learn. (...and share the pain!)

Lotus Domino TimeDate Structure

2007-06-19T12:53:00.000-07:00

This structure is used in many places within Domino, such as creation / modification times as well as database and replica ID's. Here is a diagram showing just what nuggets of interesting information are stored in this critical Notes data structure:

Analysis: "Transaction Error: Failure at Service"

2007-06-05T09:20:00.000-07:00

Had this error pop up on a handheld today when trying to send from the device, got the dreaded red X. Receiving messages, however, was fine. Usually this boils down to one of a couple of issues:

1) BES server issue
2) Carrier issue
3) Service Book issue

Since the BES was fine and the carrier seemed to be OK for other users, it must have been a Service Book issue. Tried deleting and then undeleting the Desktop [CMIME] service book, didn't work. Tried deleting the Desktop [CMIME] service book, then pushing service books from the BES. Didn't work. Since lookups were also having a problem, I theorized that the odds of 2 service books (lookups are the Desktop [ALP] service book) getting corrupted was pretty small, so time to move onto the next possible culprit.

To take the carrier out of the picture, I hooked the device up to USB so that the messages would be sent using the BlackBerry Router over USB / LAN, totally bypassing the wireless carrier and thus getting that variable out of the picture.

Well I still got the Red X. Hmmm.... but this time it did not specify "Failure at Service". This time the sub-error was "Decryption Failure". Interesting: so now we are getting down to the root of the problem and I can definitely say it is not the carrier or the service books.

So I check the Dispatcher logs, since it is responsible for encryption/decryption to the device, and voila:

[30368] (06/05 10:41:18):{0xFC0} {User} Packet has been delivered to device, Tag=4758107
[40700] (06/05 10:41:44):{0xD8} {User} Receiving packet from device, size=127, TransactionId=1429352292, Tag=259213, content type=ITADMIN, cmd=0x3
[20209] (06/05 10:41:44):{0xD8} {User} DecryptDecompress() failed, Tag=259213, Error=604
[40275] (06/05 10:41:44):{0xD8} {User} Sending transaction error to device for transaction 1429352292, size=46, TransactionId=-966799068, Tag=4758202

So now I know it is an encryption issue, so let's generate a new key:

[40700] (06/05 11:05:13.616):{0x1468} {User} Receiving packet from device, size=225, TransactionId=1429352316, Tag=7143787, content type=OTAKEYGEN, cmd
=0x3
[30222] (06/05 11:05:13.616):{0x1468} {User} MFH: contentType=OTAKEYGEN, sizeOTA=180, sizeOTW=180, TransactionId=1429352316, Tag=7143787
[30308] (06/05 11:05:13.616):{0x1468} [BIPPa] {User} Forwarding data to BES Agent (S45776843), size=218, intTag=1911024, Tag=7143787
[30311] (06/05 11:05:13.616):{0x1438} {User} Forwarding status to relay, intTag=1911024, Tag=7143787, Status=1
[40279] (06/05 11:05:13.616):{0x1438} {User} SubmitToRelaySendQ, Tag=7143787
[30222] (06/05 11:05:13.616):{0x1288} {User} MTH: contentType=OTAKEYGEN, sizeOTA=14, sizeOTW=14, TransactionId=-2070675455, Tag=6675317
[30310] (06/05 11:05:13.616):{0x1288} {User} Forwarding internal data to device, contentType=OTAKEYGEN, routing=S45776843, device=23F7A64F, size=55, cm
d=0x3, ack=0, TransactionId=-2070675455, intTag=7486040, Tag=6675317, Submit=1
[40279] (06/05 11:05:13.616):{0x1288} {User} SubmitToRelaySendQ, Tag=6675317

Well it looks like it didn't do much, let's go back to the MAGT log and see what it did during this same time period:

[40076] (06/05 11:05:13.616):{0x920} {User} SendStatusToWirelessNetworkUsingSRP SendQ, TID=1911024
[40400] (06/05 11:05:13.616):{0x920} {User} Received datagram with content type OTAKEYGEN, TID=1911024, for user User
[40386] (06/05 11:05:13.616):{0x920} {User} {User} Sending message to device, Size=49, TID=7486040, TransactionId=-2070675455
[40081] (06/05 11:05:13.616):{0x920} {User} SendToWirelessNetworkUsingSRP - SendQ, TID=7486040
[30591] (06/05 11:05:13.616):{0x920} {User} {User} *** Activation *** transaction aborted.

OK, so it looks like when trying to generate an encryption key from the device, it goes through a mini Activation, however it aborts the activation, I am guessing, because it can't establish encryption with the device. So it's a catch-22.

Let's hook it up to the USB cable and provision the device & see what happens, theoretically this should just so the same things - resend the service books and regenerate the key - but do it over the wire. Let's look at the MAGT logs:

[30615] (06/05 11:09:33.130):{0x920} {User} Received RPC command [ProvisionUser] (id=1666)
[40000] (06/05 11:09:33.130):{0x920} {User} Synchronizing user configuration
[40000] (06/05 11:09:33.130):{0x920} {User} Synchronizing device capabilities
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Updated handheld capabilities info
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Handheld capabilities indicate AES encryption is supported
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Using key type AES
[40000] (06/05 11:09:33.161):{0x920} {User} Synchronizing AES encryption keys
[30000] (06/05 11:09:33.192):{0x920} {User} *** Activation *** User activated; triggering message pre-population
[30616] (06/05 11:09:33.192):{0x920} {User} RPC command [ProvisionUser] successful (id=1666)

Everything looks OK there, and now we can do lookups and send mail just fine.

What is different about the USB provisioning? Ideally I would like to be able to fix these types of problems OTA, in case someone is out of the office. I don't like the idea of having to hook up to the BES Manager and provision via USB to fix. Anybody out there know anything I could have tried OTA before hooking up to USB, without having to go through a full wireless enterprise activation?

BES 4.x Out of Office Agent Integration Breaks when Mailfile owner set to Editor in ACL (Update: Fixed in 4.1 SP5!!!)

2007-05-30T13:24:00.001-07:00

Do you use the Out of Office functionality that is integrated into the BlackBerry? It's a wonderful feature, unless you have followed IBM's recommendations on security and limited your mailfile access to Editor.

How could this possibly affect the Out of Office agent usage from the BlackBerry you ask? Well let me tell you it took me quite awhile to figure this out. I'll start from the beginning.

The Out of Office agent is an agent that runs in your mailfile, and when enabled responds to people who send you mail while you are out. In versions of Domino prior to R6 "Manager" mailfile access was required in order to enable this agent. A specific field "$AssistFlags" in the agent was set to "E" to signify that the agent was enabled, and the "E" was removed from this field to signify that the agent was disabled.

For Domino R6 and later, however, a mechanism was put into place which allowed an organization to ratchet down user level access to Editor, which tightened up security and also reduced helpdesk calls about users deleting their own mailfiles and other ridiculous things that happen when users have too much power.

However, with Editor access users would no longer be able to enable / disable their out of office agents. A workaround was developed - a complex one which I will try not to go into here in too much detail - which allowed for the AdminP process to assign this access to the user. This involved adding a field called "$AssistFlags2" (creative, no?) to the agent.

The purpose of this flag was to tell R6 clients and above whether the agent was enabled or disabled, regardless of whether the "E" was in the older "$AssistFlags" field. New agent icons were added, so now instead of seeing only a checkmark to signify enabled or a red circle-X to signify disabled, you would now see an additional "check-5" icon.

This "check-5" icon signifies that the agent is enabled for R5 and below, but disabled for R6 and above! Yes I know this is ridiculous but that is how software coding is sometimes.

The fields would be set as follows (you can ignore the small "s" in $AssistFlags, I am not sure what that is for but appears to be unrelated to the agent status):

$AssistFlags = Es
$AssistFlags2 = D

These field settings mean that the agent is disabled for R6 and later, who know to check the $AssistFlags2 field for the real status, but enabled for prior versions that didn't know about the $AssistFlags2 field and ignored it, using the "E" in $AssistFlags.

The problem here is that the BES server acts as if it were an R5 or prior Notes client in this regard. When enabling or disabling the agent, it adds or removes an "E" from the $AssistFlags field, and completely ignores the $AssistFlags2 field.

Assuming the AdminP process has run properly the first time and setup your Editor level OOO agent access, the consequences of this are the following:

1) OOO is currently disabled, the agent icon shows a "check-5", and the fields are set to :

$AssistFlags = Es
$AssistFlags2 = D

2) A user enables the OOO agent from the BlackBerry. The BES server attempts to set the $AssistFlags field to "E", sees that it is already there, and then does nothing. The BES server does not, however, remove the "D" from $AssistFlags2, so we have the same result:

$AssistFlags = Es
$AssistFlags2 = D

The R6 Domino Mail server will use the "D" value and not run the agent, the Lotus Notes R6 client will also see the "D" value and show the agent as disabled, yet the BlackBerry shows that it is enabled. So we have a remote user who thinks the agent is activated, gets back into the office 2 weeks later, and discovers that it never sent a single message out. Bad!

Another issue comes into play as well, in the following sequence of events:

1) OOO is currently disabled, the agent icon shows a "check-5", and the fields are set to :

$AssistFlags = Es
$AssistFlags2 = D

2) OOO is enabled by the Lotus Notes R6 client, which leaves the $AssistFlags field alone but removes the $AssistFlags2 field. This change now shows the agent icon as a "check", and the fields are set to:

$AssistFlags = Es
[$AssistFlags2 field deleted]

2) A user disables the OOO agent from the BlackBerry. The BES server removes the "E" from the $AssistFlags field, however it does NOT create/populate the $AssistFlags2 field with a "D" to tell R6 clients that the agent is disabled.

$AssistFlags = s
[$AssistFlags2 field still missing]

3) The next time the user enables the OOO agent using the Lotus Notes R6 client, an interesting thing will happen, it will kick off the AdminP process once again to enable the agent for Editor level access. You see the familiar blue text and the enabling of the agent requires a delay to implement.

I believe it does this because the BES, when disabling the agent, for some unknown reason also clears the "Run on Behalf of" field, which is needed to allow the agent to run as the user:

In addition to this, when AdminP finally processes the request, grants the access, and enables the agent, the BES server does not pick up this change, for reasons which escape me at this point.

So the result of this sequence is an OOO agent that is enabled on the mail server, shows as enabled on the R6 Notes client, but shows as disabled on the BlackBerry device. This is also bad.

Here is what I believe needs to be done to resolve these issues:

1) Modify the BES server logic such that it acts like an R6 client, that is aware of the $AssistFlags2 field and modifies it instead of (or in addition to) the outdated R5 $AssistFlags field. Of course this might necessitate the minimum Domino mail server level be at R6 or higher, but since we are coming up on R8 and even R6 support is expiring, I don't see this as a huge issue.

2) Ensure the BES server logic does not clear the "Run on Behalf Of" agent field when disabling the OOO agent.

3) Ensure that changes made to the OOO agent by the AdminP process are picked up by the BES server, such that the Domino Mail Server/Notes Client and the BlackBerry device are always in sync as to the status of the OOO agent.

Until these issues are addressed, I see no easy way of programmatically blocking my users from using the OOO functionality on the BlackBerry device, which results in the above issues. My last resort is to send a mass email saying that the BlackBerry OOO feature (which people love) is broken and to always use the Notes client. Unfortunately this will result in higher helpdesk calls, since we always have people leave the office who forget to enable their OOO and end up calling us to enable it for them.

If anyone out there can shed additional light on this issue or possible workarounds, feel free to let me know. I am also actively working on this with RIM, and hope to have it recreated in their environment and logged as an SDR so that it may be addressed in a future service pack.

GPS Reporting to BES Server in 4.1 SP3 IT Policy

2007-05-30T08:40:00.000-07:00

Upon upgrading to 4.1 SP3, I noticed that the IT Policy has a new Policy Group named "Location Based Services". Under this new group are the following options:

Disable BlackBerry Maps: (Default: False, Requires: Device code 4.0.0 and higher)

Enable Enterprise Location Tracking: (Default: False, Requires: Device code 4.2.1 and higher)

Enterprise Location Tracking User Prompt: (Default: "Your Location is now being tracked at the server", Requires: Device code 4.0.0 and higher)

Enterprise Location Tracking Interval: (Default: 15 Minutes, Requires: Device code 4.2.1 and higher)

I was very excited to see these built in options, which allow tracking via GPS without having to purchase a third party solution. I enabled the tracking, leaving the interval set to 15 minutes, but changing the apparently mandatory notification to "Please call (312) XXX-XXXX to return this device." This serves the purpose of providing some sort of notification, but not letting the user know that their device location is being tracked. I named this policy "Device Tracker" so that I can use this to locate stolen devices, it is not used as a general purpose policy.

I then applied this policy to single user who uses a BlackBerry 8800 device at code level 4.2.1. Everything works great... except for the fact that I have no idea where the GPS information is logged to on the BES server. I would suspect the SQL server in a device related table, but cannot find anything.

I have asked RIM and am awaiting a response... I guess they are not sure either where this information is stored. Once this tiny issue is sorted out I am going to have some fun with Google Maps mashups, making tracking lost or stolen devices that much easier.

BES Domino 4.0 / 4.1 Interoperability Issues - Updated 6/1/07

2007-05-25T07:40:00.000-07:00

I upgraded one of my main BES servers to 4.1 SP3 HF1 last weekend, but left the other two at 4.0 SP6 HF4. Since the DB Schema was upgraded, we needed to upgrade all of our BB Manager workstations to 4.1 as well.

Now, we are seeing an issue where, using the locally installed 4.1 Manager, we cannot provision a new user added to one of the 4.0 servers using the USB cable. The service books get assigned but the rest of activation / synchronization / prepopulation does not kick off. Wireless activation of new users on the 4.0 servers also seems erratic.

BTW, this upgrade was performed after getting the word at WES that 4.0 and 4.1 servers sharing the same SQL database is OK, since they have many larger accounts than ours and they can't expect them to upgrade 10's of servers at the same time.

Well apparently it is OK for existing users but not newly added users! RIM's tech's response now is: Well it can work but it is not recommended, and the "workaround" is to upgrade the remaining servers to 4.1.

Update 6/1/07: Apparently the tech I spoke with on this issue was incorrect, RIM verifies that they fully support a mixed 4.0/4.1 environment as they told me at WES just a few weeks ago. This issue with USB activation has to do with an identified bug in the 4.1 SP3 BlackBerry Manager console, which is fixed in the upcoming 4.1 SP4, but also can be worked around by downgrading to the 4.1 SP2 Manager console. Thanks to RIM for clarifying this for me.

BlackBerry ID Numbers... MOST of them

2007-05-25T07:17:00.000-07:00

BlackBerry devices are multifunction, convergence devices. This being the case, they have multiple methods for identifying the different parts that make up the device. Here is a guide to the different serial / ID numbers used by the different components that make up the BlackBerry device:

Model: 4 digits + letter, tied to hardware, identifies hardware type (digits / letter) & carrier (letter)

PIN (Personal Identification Number): 8 Hex characters, tied to hardware, used for routing within the RIM network

IMEI (International Mobile Equipment Identity): 15 digits, tied to hardware, used to identify a
GSM device

Phone Number: Variable, tied to SIM card, used for voice calling (duh)

IMSI (International Mobile Subscriber Identity): 19 digits, tied to SIM card, used to identify a particular carrier account

FCC ID (Federal Communications Commission ID): 9 alpha digits (starts with L6A for RIM), tied to hardware, used to register wireless device with government agency

BT MAC (Bluetooth Media Access Control): 6 Hex pairs, tied to hardware, used to identify Bluetooth personal area network device

Missing Device Phone Number fields in BB Manager

2007-05-25T07:12:00.000-07:00

This is a simple question that has devolved into a mini research project ( like so many do!)

We are trying to reconcile carrier billing info with active BlackBerry users, and are finding many SIM card accounts we are paying data plans for but we don't know who is using the device or where it is.

I use the BB Manager to find devices by carrier phone number which is easy most of the time, however many accounts (229 out of 1905 or about 12%) do not have the "phone number" field populated in BB Manager, these are the accounts I am trying to track down.

Results so far:

1. I have discovered that the "My Number:" field at the top of the Phone app pulls directly from the SIM card's "Phone number" field. If present in the Phone app display, the "My Number:" field then populates the BB Manager account "Phone Number" field in the SQL database.

2. Interestingly, this field can be user edited via Options / Advanced / SIM Card / Edit SIM Phone Number. Also interestingly, it appears to be a display field only and does not change the phone number that the SIM card actually uses.

3. Normally this SIM Card "Phone Number" field is populated by the carrier during the provisioning process, and before they ship the SIM Card out. In some cases, however, they sloppily forget this step which results in a missing SIM card field, and thus an "Unknown Number" on the Phone app display, however the SIM & voice calling work just fine, as again the true number is encoded into the SIM and unrelated to the "Phone Number" SIM display field.

4. It is easy enough for the end user to edit and add their phone number if nonexistent, or change it if the number associated with the SIM is changed by the carrier.

The issue for me is that I don't want to try to contact >200 users and have them run through a manual process of adding this field with the proper number. I am looking for an automated way of pushing it out. There doesn't appear to be any hack for this, so I will have to contact the wireless carrier's and see if they have a solution.

BES SQL Permissions Explained - Part 3

2007-04-16T09:46:00.001-07:00

In this post we will discuss the necessary SQL permissions to allow administrative access to the BES database, as well as the new role based administration features in 4.1.

First, let's create a new SQL login named 'bbadmin' with no server roles or database access or database roles.

Now, what happens when we try to use this login and password from a remote BlackBerry Manager console?

[20000] (04/16 11:50:20.563):{0x150} COM Error 0x80004005 in ConnectionItem::ConnectToDB() - Cannot open database requested in login 'BESMgmt'. Login fails. - Unspecified error (connection string- -Provider=SQLOLEDB;Server=SQLTEST000;Database=BESMgmt;uid=bbadmin;pwd=)

So now let's give Permit access to the BES database.

And once again...

[20000] (04/16 11:54:34.257):{0x4E0} [ODBCRecord::DoGetFirstValue] SQL error: [0x80040E09 SELECT permission denied on object 'ObjectDefn', database 'BESMgmt', owner 'dbo'.] Source: [Microsoft OLE DB Provider for SQL Server] SQL State: [42000] NativeError: [229]

Same error popup, but the logs tell a different story. We get into the database but then have permissions issues when trying to do SQL calls. Per the RIM documentation, administrators also need the 'rim_db_bes_server' database role in order to do their work. So let's follow their advice and add that role to the account:

And voila, we are in!

Ummm, or are we? Everything is empty here. Not much to manage! Interesting that we have 'Unknown Authority' listed at the top. Although the 'rim_db_bes_server' role was sufficient in v4.0 to manage the server, apparently things have changed in 4.1.

New database roles were added in order to avoid giving full access to anyone that needed to do anything on the server. Let's take a look at them:

These are all the roles, the ones that have been added in 4.1 all start with 'rim_db_admin...'. We can also ignore the 'audit' roles which are read-only and for training purposes only, which leaves us with:

rim_db_admin_jr_helpdesk
rim_db_admin_sr_helpdesk
rim_db_admin_handheld
rim_db_admin_enterprise
rim_db_admin_security

Let's first remove our legacy 'rim_db_bes_server' role, then add the 'rim_db_admin_jr_helpdesk' role and see what happens.

Excellent, we have access now, and it even shows our authority level correctly at the top. Let's move up to the 'rim_db_admin_sr_helpdesk' role and see if there is a difference.

Hmmm, the only obvious change is in the title bar, which reflects our promotion to Sr. Helpdesk. Maybe there are some tasks now available in the menus, but nothing obvious. Let's move up to the 'rim_db_admin_handheld' role.

Now we are getting somewhere, I can see a whole new tab 'Software Configurations' available. Perhaps there are other subtasks available too. Let's go ahead and switch to the 'rim_db_admin_enterprise' role.

The only difference now appears to be the addition of the MDS service under the 'Servers' leaf of the Explorer View tree on the left. Perhaps we are now able to manage MDS with this new authority. How about we now go to the ultimate, 'rim_db_admin_security'?

Wow, now I can see everything plus the Role Administration tab. I guess now I can add people and assign/remove roles without having to use the SQL administration tools, let's try it out! Why don't we first list the administrators that are in the 'rim_db_admin_security' role, it should just be ourselves, the 'bbadmin' account.

Uh oh. I thought we had the top level of access, what is going on here? I can't even list the members of a role? Something is messed up.

Oh, wait. I seem to recall RIM saying that System Administrator (sysadmin) access on the SQL server is needed to perform role-based administration. Ummm, well that is the highest level of access available on the SQL server, so I am sure the SQL team is going to be pretty uncomfortable with that, but it makes sense that we would need this access in order to manage SQL logins and database roles through the front end of the BlackBerry Manager, which is really what we are doing after all. So let's go ahead and add 'sysadmin' server role to this account and try again.

OK, so we have full access now, at the expense of an angry SQL admin team. I guess I can forgo role based admin via BlackBerry manager, and have the SQL guys remove the 'sysadmin' server role. That just means I will have to work with them to add logins & roles in the future, so I will be sure to get on their good side!

BES SQL Permissions Explained - Part 2

2007-04-13T10:54:00.000-07:00

In the last post we discussed the SQL Server roles required to create and configure the BES database on the SQL server. In this post we will discuss the database permissions required by the login the BES server uses.

First let's go back to our DbaMgr2k console and pull up the properties of the bbdbowner login, which was the SQL login we used to create the BESMgmt database. Now take a look at the Database access tab:

We can deduce a couple of things from this dialog:

1) Our SQL login 'bbdbowner' has 'Permit' checked for access to the BESMgmt database.

2) Our SQL login 'bbdbowner' is associated with the auto-created 'dbo' user of this database. I still don't understand why you need separate database accounts linked to SQL logins, but hey, I'm a newbie at this.

3) Our SQL login 'bbdbowner' has the 'public' database role assigned. This is a built-in SQL role allowing basic database access.

4) Our SQL login 'bbdbowner' has the 'db_owner' database role assigned. This is a built-in SQL role allowing all operations on the database. I guess we were assigned this role since we created the database.

Now in most installations the same SQL login is used to:

a) Initially create the database
b) Authenticate BES servers
c) Authenticate BES admins via the BlackBerry Manager console

In order to clarify just what roles require what permissions, we will break out these separate functions into separate SQL logins. We already have the 'bbdbowner' login which corresponds to a) above, the initial creation of the database.

Let's now create a login 'bbserver' and see what permissions it needs to allow a BES server to operate. First we will give it no permissions at all:

Then we will use the BlackBerry Server Configuration panel to change the SQL authentication from the 'bbdbowner' account to the new 'bbserver' account:

Note: Using the above utility only changes the SQL login credentials used by the BlackBerry Windows services, which are located in:

HKEY_CURRENT_USER\Software\Research In Motion\BlackBerry Enterprise Server\Database

To change the SQL login used by the Domino BES task we need to also modify the 'Login' and 'Password' values located in:

HKEY_USERS\.DEFAULT\Software\Research In Motion\BlackBerry Enterprise Server\Database

And here is the result when we click the 'Test SQL Server Connection' button:

OK, I guess we need to add some permissions. Let's start with at least adding 'Permit' access to the BESMgmt database...

...and try again:

OK, now that's a little better. Let's try to start some BES services now and see what happens:

Oops, I am getting an error code 5608 when trying to start this service. Interestingly, some other services started without error. They must not need access to the SQL database. Let's take a look:

BlackBerry Alert - Started Successfully
BlackBerry Attachment Service - Started Successfully
BlackBerry Controller - Started Successfully
BlackBerry Dispatcher - Failed with error code 5608
BlackBerry MDS Connection Service - Started, then stopped a few minutes later
BlackBerry Policy Service - Failed with error code 5003
BlackBerry Router - Started Successfully
BlackBerry Synchronization Service - Failed with error code 5203

What happens when we try to start the BES Domino task? Let's see...

> load bes
> Cannot start database notification system
04/13/2007 02:36:21 PM Error initialising BES. Terminating.
04/13/2007 02:36:21 PM Shutting down Performance Monitoring and SNMP
04/13/2007 02:36:21 PM BlackBerry Mailbox Agent for Lotus Domino shutdown complete

If we look in the Event Viewer we can see many entries which refer to 'permission denied', such as this when starting the BES task above:

ReportEnvironmentDB::ReportDBSchemaVersion: COM Error 0x80040E09 - IDispatch error #3081 - Source: "Microsoft OLE DB Provider for SQL Server" - Description "SELECT permission denied on object 'ServerDBVersion', database 'BESMgmt', owner 'dbo'." - Command ""

...so I guess we need to go a little further than just 'Permit' & 'public' role access to the BESMgmt database. If we look further down the database role list, we can see that in addition to the built-in database roles, the BES database installation script created some custom roles as well. One of these custom roles is called 'rim_db_bes_server', so let's assign that role as well, since it just makes sense.

04/13/2007 02:38:40 PM BlackBerry Mailbox Agent for Lotus Domino started

Ah, that's much better. The other Windows services started up just fine too. So we can determine from our little experiment that the following minimum SQL permissions are required in order for a BES server to operate:

- SQL Server roles: None required
- BESMgmt database roles: rim_db_bes_server

Next: Minimum permissions for remote BES administration, and role-based security in 4.1

BES SQL Permissions Explained - Part 1

2007-04-12T12:10:00.000-07:00

In the last post we talked about setting up a test environment to mimic a production BES / SQL environment. In this post we will talk about the actual SQL server permissions required to create or upgrade a BES management database.

If this is an initial install, then the BES management database will most likely be automatically created and configured via the BES installation program. However, there is a pre-requisite: If using SQL authentication like we are, we need to have a SQL login defined on the SQL server with the appropriate permissions.

Using our DbaMgr2k tool, we first:

1. Double click on our SQL Server to open the Connection window. Type in the password for the default 'sa' account and click the checkbox to 'Save password'. We are using SQL authentication so you can leave the 'Trusted NT Connection' blank.

2. If you click on Logins you can see that there are only 2 users setup by default. One is the 'sa' account we used, and the other is the NT Builtin\Administrators group. Although we are using SQL authentication this account should not be removed.

3. Right click on Logins and select New Login. This new user will be the account we use in the BES installation to create and configure the BES database. Be sure to type in a password and click 'Save' before configuring any other settings; a little quirk of our free tool.

4. Click on the 'Server Roles' tab and check the 'serveradmin' role and the 'dbcreator' role. These roles give the login account permission to create new databases and do other server wide tasks. For now we can ignore the 'Database access' tab, as there is no BES database created yet. Click 'Save' and then 'Close' to save and exit.

5. Now from our BES install, we connect to the SQL server using these credentials, and we are prompted to create the database.

Just for fun, let's see what happens when the SQL login does not have the
'serveradmin' role assigned. Of course, we get the dreaded error message:

But what actually fails? Lets check the installer logs to find out:

[30000] (04/12 14:55:19.192):{0x33C} SQL being executed:
EXEC sp_addmessage @msgnum = 60002, @severity = 16,
@msgtext = N'Unable to add new BlackBerry Agent "%s" AgentID=%d Machine Name=%s',
@lang = 'us_english'

/* BESRouter errors */
IF EXISTS (SELECT error FROM sysmessages
WHERE error = 60103)
EXEC sp_dropmessage @msgnum = 60103, @lang = 'us_english'
[20000] (04/12 14:55:19.192):{0x33C} SQL Error Message from CBESDBInstaller::ExecuteSql.executeDirect: SQLSTATE: 42000 Native error: 15247 Message: User does not have permission to perform this action.

Looks like the DB installer script was unable to add custom BES error messages to the SQL server.

How about the other way around, the SQL Login has 'serveradmin' permissions but not 'dbcreator'?

[20000] (04/12 14:49:17.952):{0x12C} SQL Error Message from CBESDBInstaller::ExecuteSql.executeDirect: SQLSTATE: 42000 Native error: 262 Message: CREATE DATABASE permission denied in database 'master'.

In this case the DB installer script could not even create the BES DB and failed immediately. Looks like we need both roles in order to create or upgrade the database.

Now that we have the database created and understand the SQL Server roles required, the next post will examine the database roles required for daily server functioning and administrative tasks.

BES SQL Permissions Explained - Part 0

2007-04-12T06:55:00.000-07:00

I work for a large law firm, so each department is separate, even within IT. When we upgraded from 2.2 to 4.0 I needed to work with our SQL folks to setup and host the new SQL DB for the BES servers.

This coordination can be difficult, especially since I know little about the operation of SQL server and had difficulty communicating what I needed. It doesn't help that RIM's documentation about SQL permissions is conflicting in many cases, and dependent upon which version of 4.x you are running, as well as which components are being installed. Ugh!

This post will attempt to decode the mystery of SQL server permissions for a BES Domino environment. Here in part zero (zero b/c I don't actually talk about permissions yet!) I will discuss setting up a test environment for exploring how these parts work together to begin with.

In order to get the skinny on all of this, I had to hack together a test environment that mimicked a production BES / SQL install. I did this by:

1) Installing the free Microsoft Virtual PC software. (Free!)
2) Creating 3 Win2K SP4 virtual machines; two BES 4.0 servers and a SQL server

I used Win2K b/c there are no messy issues with online activation required, I reused some old keys I had written on the CD's. Since this is a test environment I don't feel too bad about that.

Not having access to the SQL server code or management tools, however, I had to improvise, which was an exercise in itself. From the BES 4.1 installation media, I installed the MSDE database engine (tools\SQLRun01.msi) on my "SQL" virtual server. I then performed the following to turn it into an almost gen-u-ine SQL server:

1) Change the default 'sa' password from NULL to something else

To do this you need to use the command line OSQL tool which allows you to run SQL scripts from a command prompt. For this I used the following commands:

osql -E
1> exec sp_password @old = null, @new = 'newpassword', @loginame = 'sa'
2> go
Password changed.
1> exit

This runs a Stored Procedure called 'sp_password' on the SQL server itself to change the password. Make sure to spell 'loginame' with only one 'n'!

2) Change the authentication mode from default of "Windows" authentication to "Mixed" authentication.

Note that you can have two authentication modes in MSDE / SQL:

- NT only using AD or local Windows users / groups (Windows)
- NT + SQL where logins are managed with SQL itself (Mixed)

Note that you cannot have just SQL authentication by itself.

What this change does is allow native SQL logins to access the SQL server, so I don't have to worry about ugly NT users / groups / permissions. I find SQL authentication easier and cleaner to manage and make sense of in my head. For this we need to:

- Stop the MSSQLSERVER service
- Change HKLM\Software\Microsoft\MSSqlserver\MSSqlServer\LoginMode from 1 to 2
- Restart the MSSQLSERVER service

3) Enable network access to the "SQL" server

By default MSDE is only useful for applications local to the machine it is installed on. Changing this allows remote machines on the network to access the MSDE SQL server just like it's big brother.

- Stop the MSSQLSERVER service
- Open command prompt
- Run 'svrnetcn'
- Click 'TCP/IP' and click 'Enable'
- Click 'Named Pipes' and click 'Enable'
- Click OK
- Restart the MSSQLSERVER service

4) Test login over the network.

- Copy the OSQL.EXE utility to another machine on the network
- From a command prompt, run OSQL -S [hostname or IP] -U sa -P newpassword
- You should get a '1>' prompt, meaning you are connected to the server.

OK, so now we have a working, network accessible SQL server. How do we manage it without learning all the arcane OSQL syntax and SQL stored procedures? Fortunately there is a free management tool out there called "DBAMgr2K". Google this name and you should be able to find the website and download it.

It is basically a free, simple version of SQL Enterprise Manager for use with MSDE databases. It is great because it acts and displays very similarly to the real SQL Enterprise Manager, so when I was comfortable operating this tool, I could tell the SQL guys exactly what to do in EM to get the same result in our production environment.

Next up: Permissions details for BES SQL databases

Decoding BlackBerry Server NSD Logs

2007-03-19T09:07:00.000-07:00

Even though I've tried to optimize my environment to the best of my ability, I will still occasionally see the dreaded Fault report that one of my BES servers has crashed. If this happens to you, here is how you can decode the NSD log that Domino creates and track it back to what BES was doing at the time. In some situations - not all - this can help you figure out what is going on and resolve it yourself instead of waiting for a log file analysis from RIM.

For example, I just happen to have an NSD log from a crash that happened this morning. These crash files can be found in the "Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT" directory of your data partition. The NSD log file is name something like this:

nsd_W32I_[YOUR SERVER NAME]_2007_03_18@21_15_02.log

In my experience, most of the relevant information can be gleaned from the first page full of information. The first thing we want to look for is the following line right near the top:

Arguments : "c:\lotus\domino\nsd.exe" -dumpandkill -termstatus 5 -crashpid 4592 -crashtid 3252

The most important information is the "-crashtid 3252" portion, which indicates that the thread ID that caused the crash is 3252 decimal. Well if you remember, the BlackBerry log files record the thread ID for every operation, however they are encoded in Hex. So using calc we convert 3252 decimal = CB4 Hex.

With the thread ID known, lets parse the BES MAGT log for the thread ID CB4. I use the line "grep 'xCB4' [logfile name]" The final line of this thread process is this:

[40000] (03/18 20:42:38):{0xCB4} CN=XXXX/OU=XXXX/O=XXXX!!mail\XXXX.nsf, fetching modified documents since 03/18/2007 08:42:37 PM for user XXXX

Of course identifying information has been removed to protect the innocent, however this appears to be the last thing this thread did before the server crashed. I think we have our culprit mailfile.

So now I open the mailfile and look in the All Documents view, and discover there is a Sent Item from guess what time? That's right: 03/18/2007 08:42:37 PM. Could this be our culprit document? It appears to be a reply, initiated from the device, to an email that contains a large bitmap in the body.

To verify, I check the NoteID; it is "1C542". If I now do a search on "1C542" in the NSD file I find the following lines:

@[22] 0x6001f146 nnotes._NSFNoteOpenExtended@24+678 (71,1c542,4080000,0,35b4eb34,17)
@[23] 0x6003d3b7 nnotes._NSFNoteOpen@16+55 (71,1c542,0,35b4f7d0)
[24] 0x0058ab01 nBES (6a033c,5f,71,1c542) - SYM FILE OUTDATED!
[25] 0x0058f2d4 nBES (1c542,0,0,d4683b3) - SYM FILE OUTDATED!
# 35b4f310 35b4f7d0 00000000 0001c542 0bd1a0e4 |...5....B.......|
@[22] 0x6001f146 nnotes._NSFNoteOpenExtended@24+678 (71,1c542,4080000,0,35b4eb34,17)
# 35b4f354 35b4f380 6003d3b7 00000071 0001c542 |...5...`q...B...|
# 35b4f374 0001c542 35b4f7d0 00000010 35b4f5ac |B......5.......5|
@[23] 0x6003d3b7 nnotes._NSFNoteOpen@16+55 (71,1c542,0,35b4f7d0)
# 35b4f380 35b4f5ac 0058ab01 00000071 0001c542 |...5..X.q...B...|
# 35b4f560 35b4f5c4 0058f993 0001c542 35b4f7c8 |...5..X.B......5|
[24] 0x0058ab01 nBES (6a033c,5f,71,1c542) - SYM FILE OUTDATED!
# 35b4f5bc 00000071 0001c542 00000000 35b4f7d0 |q...B..........5|
# 35b4f61c 00000002 0d44e858 00475182 0001c542 |....X.D..QG.B...|
[25] 0x0058f2d4 nBES (1c542,0,0,d4683b3) - SYM FILE OUTDATED!
# 35b4f620 0d44e858 00475182 0001c542 00000000 |X.D..QG.B.......|
# 35b4f700 0001c542 0d2e7b15 1f7957bc 00096573 |B....{...Wy.se..|
# 35b4f7d0 00003306 0001c542 00000001 00000000 |.3..B...........|
@[22] 0x6001f146 nnotes._NSFNoteOpenExtended@24+678 (71,1c542,4080000,0,35b4eb34,17)
@[23] 0x6003d3b7 nnotes._NSFNoteOpen@16+55 (71,1c542,0,35b4f7d0)
# 35b4f7d0 00003306 0001c542 00000001 00000000 |.3..B...........|
[24] 0x0058ab01 nBES (6a033c,5f,71,1c542) - SYM FILE OUTDATED!
[25] 0x0058f2d4 nBES (1c542,0,0,d4683b3) - SYM FILE OUTDATED!

It looks like NBES was trying to open this Note using the NSFNoteOpen and NSFNoteOpenExtended API calls when it crashed the server. So now I am sure this note caused the crash. What now? Delete the note from the user's mailfile so that it now resides in the trash, then restart the BES server. Of course you want to let the user know why their message is in the trash, and of course follow up on RIM on why this happened in the first place. But at least you have the server back up!

Moves between Servers - Not 100% foolproof

2007-03-16T13:26:00.000-07:00

I moved over 500 accounts from an old BES server to a new BES server today. I set the expectation with our regional managers and BB admins that there would be no impact to end users during this migration. How could I be so naive?

First, a minor amount of users were not able to send or send & receive after the move, and required either pushing the service books, reactivation, or complete account removal and re-add / re-activate to get working again. This is not so bad, it is a small number (<20 probably) of 500 people.

The bigger problem was this: The source (old) BES server's "Blackberry Synchronization Service" would sometimes freak out during bulk moves. I would move maybe 50 people at a time max, but apparently that was too much for the sync service to handle. So it would just quit, and I would have to restart it. I also noticed that the initial moves went really fast, a few per minute, but then they started to bog down and it would take 5 minutes to move one account.

That is no big deal, until I realized by looking through the logs what was happening: the sync service contains the device backup service, and it was going through and deleting the backup data not only for the accounts being moved, but FOR EVERY USER ON THE SERVER: (names removed to protect the innocent)

[46036] (03/14 10:15:41):{0xC50} [SYNC-Gate] Start removing user. [W
[46036] (03/14 10:15:41):{0xC50} [SYNC-Gate] Start removing user. [R
[46036] (03/14 10:15:42):{0xC50} [SYNC-Gate] Start removing user. [Z
[46036] (03/14 10:15:42):{0xC50} [SYNC-Gate] Start removing user. [O
[46036] (03/14 10:15:43):{0xC50} [SYNC-Gate] Start removing user. [S
[46036] (03/14 10:15:43):{0xC50} [SYNC-Gate] Start removing user. [H
[46036] (03/14 10:15:44):{0xC50} [SYNC-Gate] Start removing user. [K
[46036] (03/14 10:15:44):{0xC50} [SYNC-Gate] Start removing user. [D
[46036] (03/14 10:15:45):{0xC50} [SYNC-Gate] Start removing user. [G
[46036] (03/14 10:15:45):{0xC50} [SYNC-Gate] Start removing user. [B
[46036] (03/14 10:15:46):{0xC50} [SYNC-Gate] Start removing user. [D
[46036] (03/14 10:15:46):{0xC50} [SYNC-Gate] Start removing user. [O
[46036] (03/14 10:15:47):{0xC50} [SYNC-Gate] Start removing user. [S
[46036] (03/14 10:15:48):{0xC50} [SYNC-Gate] Start removing user. [R
[46036] (03/14 10:15:48):{0xC50} [SYNC-Gate] Start removing user. [A
[46036] (03/14 10:15:48):{0xC50} [SYNC-Gate] Start removing user. [C
[46036] (03/14 10:15:49):{0xC50} [SYNC-Gate] Start removing user. [F
[46036] (03/14 10:15:49):{0xC50} [SYNC-Gate] Start removing user. [N
[46036] (03/14 10:15:50):{0xC50} [SYNC-Gate] Start removing user. [R
[46036] (03/14 10:15:50):{0xC50} [SYNC-Gate] Start removing user. [W
[46036] (03/14 10:15:51):{0xC50} [SYNC-Gate] Start removing user. [B
[46036] (03/14 10:15:51):{0xC50} [SYNC-Gate] Start removing user. [G
[46036] (03/14 10:15:52):{0xC50} [SYNC-Gate] Start removing user. [A
[46036] (03/14 10:15:53):{0xC50} [SYNC-Gate] Start removing user. [P
[46036] (03/14 10:15:53):{0xC50} [SYNC-Gate] Start removing user. [Z
[46036] (03/14 10:15:54):{0xC50} [SYNC-Gate] Start removing user. [B
[46036] (03/14 10:15:55):{0xC50} [SYNC-Gate] Start removing user. [L
[46036] (03/14 10:15:58):{0xC50} [SYNC-Gate] Start removing user. [M

Once this happened, it kicked off an OTA device backup, which once again sync'ed everything back from each device, which slowed down EVERYTHING.

Now here is the worst part: many of the devices (with accounts that were not even scheduled to be moved that day) received either:

1) An activation complete - OK prompt
2) A continuing activation process
3) A leftover activation icon on their ribbon (home screen).

I started to get calls from admins in other cities, who weren't even scheduled to be moved that day. Bad.

Lesson: Don't move more than 10-20 accounts at a time, and watch the SYNC service.

Oh, and by the way - the brand new server hardware I migrated to? It crashed this morning. :(

Gathering Wireless Vendor Info from the SQL DB

2007-03-07T08:37:00.000-08:00

My manager asked for a report of the BlackBerry users and their associated wireless provider. Given this is all stored in the BES SQL DB, it shouldn't be a problem to get the info - but there are some things to figure out before it works right:

1. There are two fields in the BES SQL DB that refer to a wireless provider. The first is the VendorID field, which specifies the actual maker of the BlackBerry device itself, regardless of what network it is currently connected to or roaming on.

2. The second is the Home Network field, which actually doesn't make sense because it shows you which wireless network provider the device is connected to right now, so it may not actually be the home network, but a roaming partner.

For example I have a user whose device is a T-Mobile device and their home network would obviously be T-Mobile, however they currently appear as Cingular under Home Network.

To get these values out of the SQL DB we need a query, here is the one I use to pull the data into Excel:

SELECT UserConfig.DisplayName, vHandheldSummaryInfo.VendorID, vHandheldSummaryInfo.HomeNetwork
FROM BesConfig.dbo.UserConfig UserConfig, BesConfig.dbo.vHandheldSummaryInfo vHandheldSummaryInfo
WHERE vHandheldSummaryInfo.PIN = UserConfig.PIN

Note that you need to join these two tables in Microsoft Query using the PIN field, not the default "id" field that it starts with, otherwise your data won't match. Haven't discovered why, perhaps their are two different id fields in the two tables. Anyway, just use PIN because that is a common key that I know is unique.

So once you run this query you get the user's Display Name, Vendor ID, and Home Network. Now to get the device's carrier I need just the Vendor ID field but I like to see what Home Network people are on, it is interesting to see where your people are in the world and what unique names other wireless carriers have, such as "CCT Boatphone" (!). (Apparently from the British Virgin Islands, but is it really for boats? a cruise ship? hmmm...)

One final issue is that the Vendor ID is in decimal format, so I wasn't quite sure which was which. The Home Network let me see which vendor ID matched up with which vendor name, but after doing a bunch of work for that I figured out that the vendor id's are stored in where else... the Vendors.xml file!

While the Vendors.xml file did in fact contain all the matchings between vendor ID and vendor name, the id's in this file were in Hex format! :( After a little hacking in Excel with the HEX2DEC formula, I came up with the following master table below of all vendors RIM supports:

Hex (vendor.xml)	Decimal (BES DB)	Provider Name
64	100	T-Mobile US
65	101	AWS
66	102	Cingular Wireless
67	103	Nextel
68	104	Sprint PCS
69	105	Verizon
6B	107	Rogers
6D	109	BWA (Aliant/Sasktel)
6F	111	MM02 DE
71	113	Telfort
72	114	T-Mobile DE/AU
73	115	TIM
74	116	Hutchison
75	117	Bouygues
76	118	SFR
77	119	Orange_France
78	120	VODAFONE (UK)
79	121	Telcel
7A	122	Telstra
7B	123	T-Mobile (UK)
7C	124	Vodafone Germany
7D	125	MM02 UK/IRL/NL
7E	126	Telus
7F	127	SMART
80	128	Starhub
81	129	Telefonica
82	130	Swisscom
83	131	Cable and Wireless
84	132	Vodafone (IT)
85	133	Vodafone (ES)
86	134	T-Mobile (NL)
87	135	Cincinnati Bell
89	137	Vodafone Austria
8A	138	Vodafone Australia
8B	139	Vodafone Ireland
8C	140	Vodafone Sweden
8D	141	CSL
8E	142	Orange UK
8F	143	Vodafone New Zealand
90	144	SingTel
91	145	Globe
92	146	Optus
93	147	Orange Mobistar
95	149	Bharti
96	150	KPN
98	152	Proximus
99	153	Vodafone_Portugal
9B	155	BT_Mobile
9C	156	Earthlink
9E	158	E-Plus
9F	159	BASE
A0	160	Dobson
A1	161	Vodafone_Egypt
A2	162	Orange_Switzerland
A4	164	Triton
A5	165	Maxis
A6	166	Vodafone_Denmark_(TDC)
A7	167	MobileOne_(Voda_Singapore)
A8	168	Vodacom

So now I can run my SQL query and easily match up the vendor ID's in decimal with the vendor names from this table. Cool.

Components of a NoteID

2007-02-14T14:30:00.000-08:00

This is a diagram of the components of a NoteID that I put together after trying to figure out just how the State Database correlates it's entries with that of the messages in the mailfile. Of course there is no information actually in the State DB entry - it tracks it by having exactly the same Universal ID as the mailfile note itself.

Umm, ok...

Two separate Notes having the same Universal ID? Wasn't the idea that this ID was Universal, as in no other identical ID in the known Universe? Oh well...

Knife-Edge Cutover for Domino BES

2007-01-09T14:16:00.000-08:00

Here is a compilation of the steps required to do a knife-edge cutover (also called a forklift upgrade) to move a BES server over to new hardware. Disclaimer: I have not done this yet, I am compiling these steps for a future migration. So there may be steps missing or just wrong. But it may be useful as an initial starting point.

1. Build new server with new windows server name and ip address.
2. Stop Domino service on original server
3. Copy the following files to the new server:
- d:\lotus\domino\data\server.id
- d:\lotus\domino\data\mdsservi.id
- d:\lotus\domino\data\names.nsf
- d:\lotus\domino\data\certlog.nsf
- d:\lotus\domino\data\BES\*.*
4. Install Domino with same release and fixpack as original server (click Yes to any registry prompt)
5. Copy the following file from the original server c:\lotus\domino\notes.ini
6. Remove the BES task from startup in Notes.ini
7. Set Lotus Domino service startup to Manual if not already
6. Shut down original server completely and unplug from network
7. Delete original server from Windows AD
9. Change IP address on new server to old IP address
8. Change name on new server to old server name
9. Reboot server to have name change take effect
10. Disconnect network cable
11. From console, start Domino to ensure it operates properly
12. Shut down Domino
13. Re-connect to network
14. Set service to automatic and add BES to startup tasks
15. Start server and check for errors

Note: If using a new IP address, check the server doc Net Address field

BES Stability - A History

2007-01-09T14:00:00.000-08:00

Well the stability of our BES servers have been up and down over the last few years. Let me detail what has happened and where we are now.

2004 Implement BES in production. Ramp up to over 1,000 users within 1 year.

2005 BES 2.2 begins having stability issues with different types of faults, about once or twice a week. Requires a restart each time, which takes awhile to initialize given >1,000 users on the box.

10/2005 After submitting enless tickets and logfiles and dumpfiles to RIM, decided - even though monitoring said it was not an issue - to upgrade the RAM on our BES box from 2GB to 4GB. Immediately notice vast improvement - crashes virtually end overnight. Huzzah!

2006 Now have a new 4.0 server (w/ 4GB) that approaches and surpasses 1,000 users without issue.

Fall 2006 At about 1,300 users, 4.0 starts getting crashes now in either NBES or NSERVER process. Back down below 1,100 users and the server stops crashing (regularly)!

Lessons learned: 1,000 is just about the technical limit for a BES Domino box, even with 4GB of memory.

There are reasons why you would want to go below this limit for recoverability as well:

1) Restart times are less with lower users
2) Log file size (and associated searching, zipping up for RIM, etc) is smaller
3) Less time to do sourceless moves to another available server in DR situation
4) Less users out of service when server goes down
5) Maintain spare capacity to move users from other servers in case they go down

For all these reasons, I like to use 500 as an upper limit for number of BB accounts per BES 4.0 server. Yes this is much less than the theoretical 2,000 per server, but then I notice I end up sleeping better these days vs. when I didn't know how the server would act the next day...

Beware Sourceless Moves!

2007-01-09T13:34:00.000-08:00

A few months back we had one of our servers go down, fortunately it "only" had about 350 users on it. Well I thought what a great opportunity to test out the sourceless move option to our hot standby server in our Colocation site.

I moved a few users at a time to be sure it worked, and it only seemed to take a minute or so per user. So I got some confidence and started moving 10-20 people at a time. (I didn't get confident enough to bulk move more than that) All this time I was watching the logs to see when the accounts completed the move process to gauge how fast it was going. Pretty nice, actually, the move automatically updates the service books on the device with the new SRP ID, so no user intervention required, in fact it truly is seamless. Nice.

Well after a couple of hours all the accounts were moved over and I focused on rebuilding the original server. Then after awhile I noticed something odd - there were some error messages in the log I was not familiar with:

[NoteID] was not found in any of the 1 redirected folders

Well actually I have seen these before but not so many of them. After a LOT of investigation, I discovered that Sent Items were not being redirected to *any* of the devices that had been moved to the new server.

Although I had replicas of all the user's State DB's on the Colo BES server, I had assumed that I did not a backup of the BlackBerryProfiles.nsf database. Upon each move, the new BES would pull info out of SQL and create a new Profile entry in this empty database. "Great", I thought, "that creates a fresh clean profile entry, and besides, we are SQL all the way now, so these profile entries aren't really used."

Ummm.... not quite.

Creating a fresh profile DB entry for each user misses some critical information that, apparently - even in this age of the SQL DB as the master and core of our BES environment - exists only in the profile DB entries.

I will name names here. In particular, the following fields:

1) DeviceCapabilities
2) RedirectSelectedFolders

First of all, DeviceCapabilities is set to null upon a sourceless move, and RedirectSelectedFolders doesn't even exist!

What are the ramifications of this? Well for DeviceCapabilities, the BES server will note that it is set to nothing and assume that it is a pre-4.0 device, and thus will not reconcile the Sent Items from Notes onto the device, and will throw the error I noted above.

For RedirectSelectedFolders, this would be created and set to "1" if someone went into their subfolders and marked selected ones for redirection to the device. If this field doesn't exist, then the BES assumes naturally that they don't have any subfolders to redirect.

So the "seamless" sourceless moves result in all devices not getting Sent Items or any custom redirected subfolders. Not good.

Of course, reactivating the device fixes this stuff. But was I about to call 350 users and reactivate them all? Nope.

In order to fix this from the server side (which is where I prefer to be), I had to get a backup copy of the BlackBerryProfiles.nsf DB from the downed server, and script an agent to populate the fields in the new Profiles BD. Although partially scripted, this was not automated, so I ran the script for each account, copying and pasting in the long string in the DeviceCapabilities field which the script then set in the new Profiles db entry.

When these fields matched what was in the previous Profiles db, then magically people began getting their Sent Items and subfolders redirected as normal. Not a fun day for me though.

Of course I called RIM to ask about this "bug" and was told that they knew about this behavior but it really wasn't planned on being fixed.

The upshot is.... never allow the BlackBerryProfiles.nsf database to be recreated, either by deleting it yourself or doing a sourceless move to a new server. You will lose valuable info in each document. Instead, in addition to the backup replicas you have of the state DB's, have a backup replica of this DB as well, so that in an emergency you can copy and paste the profiles into the database on the backup server, so they don't get autocreated and inherit all the fields with valuable goodies in them.

One final note, this only applies to sourceless moves - if you have the original BES server up and available during an account move, the destination server will pull all the information from the original profile document into it's database cleanly.

BlackBerry Lookup - The Details

2007-01-09T13:16:00.000-08:00

The BlackBerry lookup feature allows one to do a lookup against the Domino address book from the device. Some common questions I see regarding this feature are:

1) What fields does it pull from?

and...

2) Can I customize the fields it uses?

Well the answer to #2 is No, not as of September 2007, the code is fixed and will probably not be customizable by us BES admins for awhile, if ever. The answer to the first question, however, is a little more straightforward.

Here is a chart of the fields that, if populated in the Domino address book Person documents, are delivered as results to the device:

Here is a more graphical view of the actual Person document form and matching results fields, for those visual thinkers out there:

The Purpose of this Blog

2007-01-09T13:12:00.000-08:00

This blog was created to share information related to the BlackBerry Enterprise Server for Domino software product. I am currently an administrator for 4 BES servers serving over 1,800 devices worldwide, so I have a lot of day to day experience with this product which you just can't get reading the manuals or attending the tradeshows. I hope to provide useful information related to the administration of BES servers, as well as gain some valuable feedback as well.

Thanks for browsing!