Software Configuration Push Problems cont'd...

Another issue I discovered is the following:

1. I audit all of our devices to ensure the memory is above 3MB before pushing the software, to avoid causing their devices to delete all their mail. How to do this is another post in itself!

2. We send these users emails on how to clean up the device.

3. I got some emails back saying they had plenty of memory free already!

4. I checked and found that the Device information had not been updated in 6 months or more! I was working with old data for some users!

Here is my theory:

Once you get to a very small amount of memory on the BB, the Handheld Agent will stop reporting back to the server it's details on a daily basis. Fortunately there was an easy fix for most of these guys, and that is to set "Automatic BlackBerry Device Management Enabled" to FALSE, wait a few minutes, then set it back to true. What this does is delete all the device information for that user from the SQL database, then when you turn it back on it refreshes it directly from the device. If there is enough memory available you will get new info. If memory is still low you will not get any info back.

Interesting side note: The refreshed device information will only reflect when the Handheld Agent last collected the data, not the "right now" state. So if you get todays info it may be from 5 or 7AM, not from 11AM when you changed the settings.

Software Configuration Push Problems (SCPP's)

Ok here I am again after a long bout of laziness. Anyway I am working on deploying a piece of software to all our devices and am running into fun issues. Here is one of them:

8330 Curves on Verizon running 4.2 or 4.3 OS, for some reason, do not publish their hardware details to the BES. In fact they don't even publish it if you are connected over USB to the device via BlackBerry Manager. The Hardware ID is just 0x00000000.

This is a problem because when you push software it checks the device model against the device.xml file in the AppLoader directory above the actual software package. If it doesn't find it then you get an error in the POLC log about HHCM_DEVICE_NOT_SUPPORTED and to Update your device.xml.

The issue of course is not the device.xml, but that the Hardware ID is not there for this device.

I found that upgrading from 4.2 or 4.3 OS to 4.5 will publish all the gory hardware details and software pushes will work again.

I wonder if it is stupid Verizon wanting to hide the hardware from everyone but themselves (like they do with GPS) or just an bug in their version of OS 4.2 / 4.3. Either way seems it is resolved in 4.5.

ITAdminQueue - EXPOSED!

A pretty cool new feature appeared in 4.1 SP5, resulting in a new tab called Software Configuration Status within the BlackBerry Manager application:


This tab shows a view of the previously hidden ITAdminQueue table in the SQL database, which contains all sorts of back end commands that are queued up / in process to the BlackBerry handhelds.

The view above results from setting the software configuration of my test BlackBerry to a pre-configured RSA SecurID application, turning the wireless off, and manually pushing it out via the "Deploy Software" command. You can see there are two entries listed under this new tab, which don't show a whole lot of information. I assume because they are simply queued, and have not errored out yet.

Here are the corresponding ITAdminQueue rows in the actual SQL table:


Note that the first entry is actually listed with a Command column number of 129. This corresponds to the BES Policy service's SEND_APC_ITPOLICY command. The IT Policy is always sent first whenever an application is delivered, as it includes any software application policies you have created. As these policies need to be applied before the app is installed, sometimes there are errors applying the IT policy - which can prevent an app from being delivered successfully.

Note also that this IT Policy command does not show up in the Software Configuration Status tab, which leads me to believe this tab is built using a filter to only show software application entries, not the whole table.

Below this row there are two additional rows listed, both showing the Command column set to 128. 128 corresponds to the SEND_APC_APP command, so these are the commands to send the application itself. Note that these rows in the SQL table do not have a Time Sent value, but the IT Policy does. More proof that the app will not be sent until the IT Policy is applied successfully.

One interesting thing is that the app I was pushing contained only one .cod file, so why two SEND_APC_APP commands / rows? Looking into the Policy log we find something interesting:

Queuing the package RSA SecurID, consisting of 1 modules in 2 parts.

So I think this this one .cod module is split into 2 parts for transmission to the BlackBerry, for what reason I do not know. Perhaps it was too big to send all at once?

Although this new tab of information is helpful, I see it does not allow you to delete any queued commands. Some software push issues have been resolved by running the following SQL script:

osql -S [SQLServerName] -d [DBName] -U [SQLLogin] -P [SQLPassword] -Q "delete from ITAdminQueue where (command = 128 or command = 129) and status = 6"

This command will clear all SEND_APC_ITPOLICY and SEND_APC_APP entries with a status of 6, which designates an error; this allows future pushes to proceed properly.

Some other BES Admins have recommended just deleting all rows referencing a particular PIN number to resolve issues with an individual BlackBerry not receiving an application properly. I cannot vouch for this method however, as I have not used it. I do occasionally run the SQL script above during standard maintenance of our BES environment, or if a particular application push problem occurs.

From RIM's documentation, here are some error codes you might see in this new tab:

* Device timed out waiting for module
* Device reported insufficient memory to install module
* Device reported insufficient privileges to install module
* Device reported Invalid Version in packet, supported version is %s
* Device reported that there was an Incomplete Module
* Device reported that the Module Save Failed
* Device reported a general failure installing the module
* Incomplete ACK data for APPD request
* Device reported a %s error while installing module
* Device Reported Data Format Error in packet while installing module
* Device Reported Invalid Command while installing module
* Device Reported Insufficient Body Data while installing module
* Device Reported Invalid Module Hash while installing module
* Device Reported Invalid App Data Length while installing module
* Device Reported Insufficient App Data while installing module

You can check out the support article, KB15470, for more information on these messages and there resolutions.

I am sure that this new tab will help in resolving future application push issues, however I am also sure that some manual cleaning of the ITAdminQueue table will still be needed occasionally as well!

Heading to WES 2008

Well I know some of you who read this blog will be at WES this year, looking forward to meeting all of you in addition to the great bunch of BES Admin folks that convened last year.

Generally our meeting place was in the right rear of the solutions showcase but we may also be hanging out around the BoxTone lounge, check out the WES 2008 thread on www.blackberryforums.com BES Admin corner for the latest details...

Hoping to see you there!

Address Book Contact Labels and the BlackBerry

Had an interesting issue crop up today with an end user who changed the contact labels associated with an address book entry. Here is the default when you click the Edit Contact Labels button within your address book entry form:


This person had modified the labels to reflect different information, for example changing the Pager label to Home Phone #2. The problem is that the label change does not sync over to the BlackBerry, which shows the Label still listed under Pager.

I called RIM and verified this is the way it should work, as the underlying field name does not change, and RIM maps to the field name, and not the changeable display name.

I understand the issue and agree with RIM's assessment, however this is one of those gray areas where it is difficult to determine who is responsible for the issue. Is it Lotus for providing the (perhaps unwise) ability to change display names to something that is completely different from what the underlying field name reflects? Or is it RIM for not supporting this Lotus Notes feature, even though it may not be the best of features to have from a data integrity standpoint.

As is generally the case, it is perhaps a little of both, which is why these things tend not to get fixed as each side points to the other and neither will take the responsibility to address it.

This is not that big of a deal to me, and personally I agree with RIM's assessment that their sync engine should map to the field name and not a nebulous display name that can be changed at will by the end user.

However, this is just a small example of the types of conflicts that can happen when you have two separate software entities interacting with each other, creating a no-man's land of responsibility with the customer trapped in the middle.

IT Policy Removal Options

For security reasons, the IT Policy assigned to a device cannot be removed, even if the device is wiped. It is like a permanent tattoo. Like tattoos, however, they can be removed with a long and painful procedure. At least that is the way it used to be, nowadays we have more options. Let's go over them.

1) Wipe the device. Although the IT policy still exists on the device, the wipe process opens a "window" where an activation against another corporate BES server can overwrite the IT Policy with the new one. So for moves between organizations, you can only replace one policy with another, you cannot overwrite. For people who purchase previously corporate BlackBerry's on EBay for personal use this has been quite a hassle, as they are stuck with the old corporate policy which may be locked down significantly.

Which leads to...

2) Use the policy.bin file to clear the policy. This is a long and messy procedure (which I will not get into here) which mostly gets rid of the old policy, using the pre 4.0 method of a policy.bin file and Desktop Manager. I don't like this method because of the time it takes, and the lack of being in a known consistent state at the end. It is not really "just like new" when this is done, but can be enough for most people.

3) This is a new one. With a device OS later than 4.2.2, you can use the "Remote Wipe Reset to Factory Defaults" procedure documented in an older post here:

http://besdomino.blogspot.com/2007/10/remove-it-policy-new-feature-of-41-sp4.html

4) This is also a new method, which does not require an IT Policy assignment or a BES server at all, just the Javaloader utility in the RIM developer tools download. Note that an OS of 4.3.0 or higher is required, and the Javaloader utility itself must be the latest version.

Strangely, the utility does not list version numbers, so you just have to look for the available option:

resettofactory
Reset IT policy to factory settings

...when you run the executable without parameters to see if it supports it. If it does, then this command will wipe the IT Policy right off, and set it back to default:

c:\javaloader -u resettofactory

The future of IT policy removal is a bright one, indeed.

Multiple BlackBerrys per Mailfile?

Answer: Yes with some caveats.

First, you need to have fully separate BES environments, i.e. at least 2 SQL back end databases. This is easy for us as we have an EU BES environment on it's own SQL database.

If we need to set up an additional handheld for an end user, whether they need it temporarily for an assistant, or need to test a new device, then we add their account to the other BES environment and activate their device as normal. Since BlackBerrys are international devices there is no problem getting back to the device here in the U.S to finish activation.

Notes:

1) BlackBerry activation can take considerably longer due to the WAN if you are using another continents environment like we are. Don't freak out if it takes a few hours.

2) If you are doing OTA activation, temporarily disable the original BES account as it might pick up the activation email and discard it, leaving your remote BES server waiting forever. [BES servers are selfish and will NOC-block you if you let them. :)]

3) If these requests becomes common, you might want to think about a secondary local BES environment, perhaps running with an integrated MSDE, to support this service. Of course you need an additional SRP to do this.

4) Remember that these accounts cannot be failed/moved over to other local BES servers in a DR scenario, as they are on a separate database.

5) Don't forget about these "ex-pat" accounts if on a foreign BES server. The local admins may not ever delete them b/c they don't know who they are. So you should keep track of them and delete them when it looks like the service was cancelled or otherwise abandoned.

Other than these caveats, I have not seen any issues with this in our environment.

Managing the BlackBerry Device Firewall Remotely... Possible?

Over the past few weeks as I was playing with setting device options remotely, I made a few mental notes when encountering databases (i.e. configurations) backed up to the SQL db. One of these was 'Firewall Options'.

I particularly noted this one because I have been looking for a solution to the firewall popups that appear on our user's devices. Overall of course the firewall is a good thing, but I like to keep the experience of our users seamless whenever possible, and incomprehensible (to the user) firewall popups do not make for a seamless experience.

One example of this is the RSA SecurID software for our BlackBerry devices, which we deploy so our users do not need to carry around a hardware token. The first thing this software does when installed on the device is listen for a software token which is wirelessly pushed to the device via a separate app.

This listening process constitutes a 'server' on the device, so upon installation we see the following firewall popup:



Now if I am pushing this software from the server to a device out in the field, then based on this screen the end user will be confused as to what to do. 80% of the time they may choose correctly and scroll down to click allow, but that other 20% may result in a support call.

So is it possible to:

1) Set the 'Allow' rule in the device's 'Firewall Settings' database in SQL
2) Push the 'Firewall Settings' configuration down to the device
3) Push the application and have it immediately begin listening for a token without user input?

Let's find out!

First we will monitor the SYNC log again to see if anything gets backed up to the server when an option on this firewall prompt is selected.

The first thing I notice is that if I leave the "Don't ask this again for:" box unchecked, then nothing happens... I see no log entries on the server side. If I reset the app and enable the listener again, this time choosing to check the box, however:


... voila! I do see something coming through to the server:

[60000] (02/11 11:50:51.111):{0x1710} [ODBCRecord::DoSetValue] DATA = "DbVersion[0] Data[0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000] UserConfigID[17] DatabaseName[Firewall Options] UID[1] ".

This tells me that my hunch was correct, and that any saved settings are stored in the 'Firewall Options' configuration, and therefore backed up to the server. Good news!

So let's now reset everything and query the 'Firewall Options' configuration to see the default value as compared to the updated settings:

0x1100000005000004000003000002000001000000 (Default)
0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000 (RSA Allowed)

It looks like the first numbers were changed from 11 to 27, then a bunch of numbers were inserted, extending the configuration string. I am guessing that this new set of numbers uniquely identifies the SecurID server application and the 'Allow' setting I chose.

If we reset the app, then choose 'Deny' here is what the string becomes:

0x270000010107536563757249440001046874747000060000000005000004000003000002000001000000

Everything is the same except for a single number, the '2' about 2/3 of the way through the string has changed to a '6'. So perhaps a '2' means allow, and a '6' means deny. Let's try clearing everything out, and then pushing the 'allow' string to the device before the software push:

use BESConfig
UPDATE SyncBackupRestore SET Data=0x270000010107536563757249440001046874747000020000000005000004000003000002000001000000 WHERE UserConfigID = 17 AND DatabaseName = 'Firewall Options'
UPDATE SyncConfig set SyncType=1 WHERE UserConfigId = 17 AND SyncDataSourceId=11109
go

Now we reload the application, and...


Bummer.

Guess I'll have to keep digging on this one...

[Update 2/13/08]
I have been gently reminded of the use of application control policies by a fellow blackberryforums.com member, ACJones. By setting a policy on the software configuration for the RSA SecurID software, and modifying the 'Internal Network Connections' option from 'Prompt' to 'Allow', you can avoid the initial firewall prompt completely upon OTA installation!



I have tested this and it works flawlessly. Thanks AC!

Managing BlackBerry Configuration Settings Remotely - Putting it all Together

Now that we know how to update the device settings, let's see how it works in a real world scenario. I have just nuked, reloaded, and reactivated my test 8700g with 4.2 OS - here is the before shot of the relevant configuration screens:


Now let's push down some common changes to our device settings:

1. [Messages Options / General Options] Hide Filed Messages = Yes (default = No)
2. [Messages Options / General Options] Hide Sent Messages = Yes (default = No)
3. [Messages Options / General Options] Keep Messages = Forever (default = 30 Days)
4. [Messages Options / Email Reconciliation] Delete On = Mailbox & Handheld (default = Handheld)
5. [Calendar Options] Initial View = Week (default = Day)
6. [Calendar Options] Keep Appointments = Forever (default = 60 Days)
7. [Address Book Options] Sort By = Last Name (default = First Name)
8. [Options / Date/Time] Time Zone = Central Time (-6) (default = Casablanca (GMT))
9. [Options / Screen/Keyboard] Backlight Timeout = 2 Min. (default = 30 Sec.)

These modifications will require the following sequence of SQL updates (you will need to replace the '15' UserConfigID with your target account's UserConfigID in your SQL DB) :

use BESConfig
UPDATE SyncBackupRestore SET Data = 0x550000010101000001010101000301010800014465736B746F70000A000
2543336323538313330000100030101000A00020005FFFF01000601020008
01000100090102000B000002000C000002000D000001000E00000000
WHERE UserConfigID = 15 AND DatabaseName = 'Message List Options'
UPDATE SyncBackupRestore SET Data = 0x270000000000001C02FC0300000000010000000F0000000101000500000
00100FFFF0000000F00000001 WHERE UserConfigID = 15 AND DatabaseName = 'Calendar Options'
UPDATE SyncBackupRestore SET Data = 0x0A000001010100000000000100 WHERE UserConfigID = 15 AND DatabaseName = 'Address Book Options'
UPDATE SyncBackupRestore SET Data = 0x040001000000000400021400000001000302 WHERE UserConfigID = 15 AND DatabaseName = 'Options' AND UID = -447168820
UPDATE SyncBackupRestore SET Data = 0x1000019A71C67ACBC67AB400000002000000000E00018131D8CDC28BB8
B90000000700000D0001D30CB969B820BFDD0000000600100001C2C1986B
11CE32B000000002000000780D0001BDAAFA49FFAD48CC00000006011000
013E3CBC03163D39CF00000002000001C24F0001CB2088784AFC39CC0000
000000416A6176613A2F6E65742E72696D2E6465766963652E7468656D652
E626264696D656E73696F6E5F7A656E5F333230783234305F772E5468656D
65466163746F72790D0001416028234B233EA70000000600100001F5006E1
93847CE260000000200000001100001BD91E3D465E4816900000002000000
00100001EBB9DE74CE5F773800000002000000322600011F0091A81B93E1C
30000000000186E65745F72696D5F6170706C69636174696F6E5F6D656E75
08000138D4D5F7DC80C5472E0001DA36AA198C1233A40000000000206E65
745F72696D5F62625F70726F66696C65735F6170702E50726F66696C65734
F00019B05812B5A1A15DC0000000000416A6176613A2F6E65742E72696D2
E6465766963652E7468656D652E626264696D656E73696F6E5F7A656E5F33
3230783234305F772E5468656D65466163746F72790D0001F33AFBD074DB0
7F100000006001000016B506A6FCD9D596A00000002000000011000012ECB
C575BEF57F3E00000002000000C80D0001094AC736D16697C800000006001
000011762E2440C4D5D510000000200000064080001E8D6CB484814CA60
WHERE UserConfigID = 15 AND DatabaseName = 'Options' AND UID = -1321243196
UPDATE SyncConfig set SyncType=1 WHERE UserConfigId=15 AND SyncDataSourceId=11109
go

This will kick off a one way sync from the server to the device, pushing all the configuration databases managed by the Backup component. It will take less than a minute for the SYNC process to pick up on this change, and about another 30 seconds or so to re-sync all the settings. [sleep 120] Now back to our handheld... it appears that all changes (in green boxes) have been picked up:


Now we must be sure to run the following command to restore the device to server synchronization:

UPDATE SyncConfig set SyncType=3 WHERE UserConfigId=15 AND SyncDataSourceId=11109

If we didn't do this, then any configuration updates from the handheld would not be backed up to the server. Actually, this might be an interesting option to employ for some firms who want total control over the handheld... one could schedule a daily push to refresh settings to the organizational standard settings to discourage end-user customization.

All of these commands could be easily scripted by putting them in a text file such as 'PushSettings.sql' and using the OSQL command's -i switch, i.e.:

osql -S sqlservername.company.com -U sqlaccountname -P sqlpassword -i PushSettings.sql

So that's it... We've gone down the rabbit hole into the depths of the BES SQL DB in order to discover some of it's secrets, and we came away with a method of pushing normally hidden device configuration settings over the air. Have fun, but remember to backup your SQL database regularly and TEST TEST TEST your scripts outside of production first!

Managing BlackBerry Configuration Settings Remotely - Part Five

Here are the options for the Calendar, first with default settings, then with the individual options enabled along with the resulting Data string:


[Default Settings Above]
0x270000FFFFFFFF1C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[Initial View: Week]
0x270000000000001C02FC030000000001000000
0F0000000101000500000001003C000000000F00000001

[Initial View: Month]
0x270000000000001C02FC030000000001000000
0F0000000201000500000001003C000000000F00000001

[Initial View: Agenda]
0x270000000000001C02FC030000000001000000
0F0000000301000500000001003C000000000F00000001

[Initial View: Last]
0x270000000000001C02FC030000000001000000
0F0000000401000500000001003C000000000F00000001

[Enable Quick Entry: No]
0x270000000000001C02FC030000000001000000
0F0000000000000500000001003C000000000F00000001

[Default Reminder: None]
0x270000000000001C02FC030000000001000000
FFFFFFFF0001000500000001003C000000000F00000001

[Default Reminder: 0 Min]
0x270000000000001C02FC030000000001000000
000000000001000500000001003C000000000F00000001

[Default Reminder: 1 Hour]
0x270000000000001C02FC030000000001000000
03C000000001000500000001003C000000000F00000001
(Minutes in hex: 3C = 60 decimal)

[Default Reminder: 1 Week]
0x270000000000001C02FC030000000001000000
060270000001000500000001003C000000000F00000001
(Minutes in transposed hex: 2760 = 10080 decimal ; 10080/ 60 = 168 hours / 24 = 7 days)

[Snooze: None]
0x270000000000001C02FC030000000001000000
0F000000000100FFFFFFFF01003C000000000F00000001

[Snooze: 1 Min]
0x270000000000001C02FC030000000001000000
0F0000000001000100000001003C000000000F00000001

[Snooze: 10 Min]
0x270000000000001C02FC030000000001000000
0F0000000001000A00000001003C000000000F00000001

[Snooze: 30 Min]
0x270000000000001C02FC030000000001000000
0F0000000001001E00000001003C000000000F00000001

[Start of Day: 8 AM]
0x27000000000000E001FC030000000001000000
0F0000000001000500000001003C000000000F00000001
(Minutes past midnight in transposed hex: 01E0 = 480 decimal ; 480/60 = 8 hours)

[End of Day: 6 PM]
0x270000000000001C0238040000000001000000
0F0000000001000500000001003C000000000F00000001
(Same as above, 0438 = 1080 decimal ; 1080/60 = 18 hours or 6 PM}

[First Day of Week: Mon]
0x270000010000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[First Day of Week: Tue]
0x270000020000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000001

[Confirm Delete: No]
0x270000000000001C02FC030000000000000000
0F0000000001000500000001003C000000000F00000001

[Show Free Time in Agenda View: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000000010F00000001

[Show End Time in Agenda View: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000000000F00000000

[Keep Appointments: 15 Days]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001000F000000000F00000001

[Keep Appointments: Forever]
0x270000000000001C02FC030000000001000000
0F000000000100050000000100FFFF0000000F00000001

[Wireless Synchronization: No]
0x270000000000001C02FC030000000001000000
0F0000000001000500000000003C000000000F00000001

[Show Tasks: Yes]
0x270000000000001C02FC030000000001000000
0F0000000001000500000001003C000100000F00000001


Now for the Address Book:


[Default Settings Above]
0x0A000001010000000000000100

[Sort By: Last Name]
0x0A000001010100000000000100

[Sort By: Company]
0x0A000001010200000000000100

[Confirm Delete: No]
0x0A000001000000000000000100

[Allow Duplicate Names: No]
0x0A000000010000000000000100

[Wireless Synchronization: No]
0x0A000001010000000000000000


And finally, Tasks:


[Default Settings Above]
0x0F000000000000000000000101FFFFFFFF01

[Sort By: Priority]
0x0F000001000000000000000101FFFFFFFF01

[Sort By: Due Date]
0x0F000002000000000000000101FFFFFFFF01

[Sort By: Status]
0x0F000003000000000000000101FFFFFFFF01

[Confirm Delete: No]
0x0F000000000000000000000001FFFFFFFF01

[Snooze: 1 Min]
0x0F0000000000000000000001010100000001

[Snooze: 10 Min]
0x0F0000000000000000000001010A00000001

[Snooze: 30 Min]
0x0F0000000000000000000001011E00000001

[Wireless Synchronization: No]
0x0F000000000000000000000101FFFFFFFF00

Managing BlackBerry Configuration Settings Remotely - Part Four

Now that we know how to update the settings on the device, here is a screenshot showing the default settings, and a list of Data strings with all the different Message List 'General Settings' options:


Default Message List / General Options settings:


[Data String for Defaults above]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Time: NO]
0x350000000101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Name: NO]
0x350000010001000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Display Message Header On: 1 line]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B010002000C000002000D000001000E00000000

[Display Message Header On: 2 lines]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B020002000C000002000D000001000E00000000

[Display Message Count: None]
0x350000010101000001010001000301010200051E0001000600020008
00000100090102000B000002000C000002000D000001000E00000000

[Display New Message Indicator: No]
0x350000010101000001010001000301010200051E0001000600020008
01000100090002000B000002000C000002000D000001000E00000000

[Confirm Delete: No]
0x350000010101000001010000000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Hide Filed Messages: YES]
0x350000010101000001010101000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Hide Sent Messages: YES]
0x350000010101000001010001000301010200051E0001000601020008
01000100090102000B000002000C000002000D000001000E00000000

[Make PIN Messages Level 1: No]
0x350000010101000001010001000300010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Auto More: No]
0x350000010101000001010001000301000200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 15 Days]
0x350000010101000001010001000301010200050F0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 2 Months]
0x350000010101000001010001000301010200053C0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 3 Months]
0x350000010101000001010001000301010200055A0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 4 Months]
0x35000001010100000101000100030101020005780001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 5 Months]
0x35000001010100000101000100030101020005960001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: 6 Months]
0x35000001010100000101000100030101020005B40001000600020008
01000100090102000B000002000C000002000D000001000E00000000

[Keep Messages: Forever]
0x35000001010100000101000100030101020005FFFF01000600020008
01000100090102000B000002000C000002000D000001000E00000000

[SMS and Email Inboxes: Combined]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C010002000D000001000E00000000

[SMS and Email Inboxes: Separate]
0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C020002000D000001000E00000000


The first line contains all the default settings from my test device which is an 8700g with the 4.2 OS software. The latter lines are the defaults with only the option shown enabled. The highlighted numbers in the rows show which number controls the setting in that row.

Keep in mind that this may be different from what you see depending upon your device type and OS version, however I suspect they would be similar if not the same for the same OS version.

Next: Calendar, Address Book, and Tasks options database settings.

Managing BlackBerry Configuration Settings Remotely - Part Three

To begin, of course, a Disclaimer: The BES SQL database is the heart of your BES environment. Making changes to it outside of the UI is not supported by RIM and can do irreparable harm to your entire BES environment. The following is for informational purposes only, and I am not responsible for the thousands of midnight telephone calls you will get from your BlackBerry users when an errant SQL command has rendered your BES servers mere hunks of plastic and metal, and transmogrified into a CLM [career limiting move]. I for one will not be implementing these findings in my production environment! Now, onto to the madness...

Since we have discovered where the configuration setting "Hide Sent Messages" lives in the SQL database, let's try to modify it and get it pushed out to the device. First, let's reset the option back to the default of "No" and restart the wireless to get it changed in the SQL database. Here is the device screenshot showing the default settings:


And here is the SQL data, showing that the '1' has been changed back to a '0':

0x350000010101000001010001000300010200051E0001000600020008
01000100090102000B000002000C020002000D000001000E00000000

Now, let's use a SQL command to update this string to change the '0' back to a '1':


Perhaps if we restart the wireless on the device the change will sync down?



Alas and alack, the change did not come down. Oh well, that would have been too easy. Let's take a deeper look into the SQL tables to see where we can go from here. I see a promising table named SyncConfig...and it appears to... wait for it... hold the Synchronization Configuration settings! Just what we need:

I see there are many lines associated with UserConfigId 9, which is my single test user. I also see a column titled SyncDataSourceId, I wonder what the numbers in this column refer to?

After doing some digging I discover the SyncDataSource table, which contains mappings between the numbers in the SyncDataSourceId column and the names of these sources.


What we are interested in would be the Backup data source, or number 11109, as this is what we are working with. This leads us to row 102 in the SyncConfig table.

Now the question becomes, what do the other columns in this table mean, and what can we do with them? SyncType and ConflictResolution appear especially intriguing given what we are trying to accomplish here.

By changing some settings on the device and watching the effect on this table for other unrelated rows, I discovered that the SyncType field has three options available:

1 = Server to Device
2 = Device to Server
3 = Bi-directional

In our case, the 11109 Backup row in this table indicates '3', or Bi-Directional sync. I would have expected to see this set to '2' for one way backups from device to server only. However '3' may make sense, as the backup is from device to server for the most part, but can be from server to device when changing devices for instance.

Similar to SyncType, the ConflictResolution column appears to have two options available:

0 = Server Wins
1 = Device Wins

Our row 102 shows a '0' in this column, which again is a little unexpected as I assumed that the server side was a backup to what is on the device, so the device should always win in a conflict. Perhaps there are good reasons for these non-obvious settings that I am not aware of.

In any case, we have still updated the data in the SyncBackupRestore table but it has not sync'ed to the device, even though the sync settings shown above indicate that a) sync is bidirectional and b) the server will win on a conflict. So really this should just work without any changes, however we need to figure out some way of 'kicking' off a sync.

We know that when you disable and then re-enable synchronization from the device or BES manager for other functions such as Address Book, it will kick off an initial sync. The issue here is that there is no way through the GUI of the device (or the BES Manager) to kick off a Backup sync, as this is an underlying process not available to the end user or administrator.

Why don't we try to just flip the ServerEnabled setting from 1 to 0, and then back to 1 to see if this does it:


Unfortunately, looking at the logs it doesn't appear that this trick did anything. I see the following lines in both the SYNC and CBCK logs:

[46077] (01/31 13:09:36.021):{0x1178} [SYNC-Gate] Finished user config change check. No change detected.
[40039] (01/31 13:20:21.856):{0x1EE0} CheckResponseRcvd - no changes detected for any users.

Let's move on and try to change another field, how about we try to change the SyncType from '3' (bi-directional) to '1' (server to device):


Wow! Within a minute or so of making this change there was a flurry of activity in the SYNC and CBCK logs. I notice this line as well:

[46078] (01/31 13:26:53.092):{0x1178} [SYNC-Gate] Finished user config change check. Change detected.

Turns out this triggered a full synchronization, such that everything on both sides was compared and the changes pushed down to the device. Going back into our Message List Options screen, we are presented with:





As Dora would say, "Lo Hicimos! We did it!"






So in summary, we were able to push this change to the device by doing the following:

1) Locating the option in the data string of the particular settings database within SyncBackupRestore
2) Updating the data string with the particular option set, via SQL commands
3) Modifying the SyncType from '3' to '1'

Now, like a good boy scout, we must remember to clean up after ourselves. To get this to work we had to break the default SyncType, so let's set it back to '3' so that any future changes on the device continue to be backed up:


Next Up: Decoding the Data string in the "Message List Options" database to map out all the different option settings.

Managing BlackBerry Configuration Settings Remotely - Part Two

OK so let's dive in and see what we can do with the SQL table SyncBackupRestore. First of all it looks like there are separate entries for different types of settings. Let's start with the entry for Message List Options for UserConfigId #9, which is the only user in this test environment:


It looks like the Data column is encoded in some sort of binary format, let's see if we can use OSQL to see what's in there:


So after a bunch of header information we finally see the data string, and it looks like this:

0x350000010101000001010001000301010200051E0001000600020008
01000100090102000B000002000C000002000D000001000E00000000

As it exists now, this string represents the state of the options for the Message List, i.e. the Messages screen on the BlackBerry. The best way to discover what pieces control which device settings is to change an option on the device, wait for it to get synced up to the server, then check this string again to see what is different.

We will first test by changing a single option on the BlackBerry, Hide Sent Messages from the default of No to Yes (a common enough request!). Now the issue is waiting for the device's backup agent to push the change to the BES server, however I discover to my pleasant surprise that turning off the antenna and turning it back on will trigger the handheld agent to push any device changes up to the server. (Restarting the BlackBerry has the same effect).

I have been using the Unix "tail" utility to follow the BES server's CBCK log file (Backup connector), which is set to the debug logging level of 7 through the registry (4 is the max via the UI). When the device connects to the network we see the following update come through:

[60000] (01/30 14:26:07.086):{0x151C} [ODBCRecord::DoSetValue] DATA = "DbVersion[1] Data(first 32 bytes)[0x350000010101000001010001000300010200051E000100060102000801000100] UserConfigID[9] DatabaseName[Message List Options] UID[1] ".

This log entry only shows the first 32 bytes of the updated field, so we have to go back to the OSQL utility to display the full field again, and it is now set to the following:

0x350000010101000001010001000301010200051E0001000601020008
01000100090102000B000002000C000002000D000001000E00000000

The only change in this string is the highlighted '1' near the end of the first line has now been changed from a '0' previously. This leads us to believe that this number controls the Hide Sent Messages option on the device, and that modifying it from '0' to '1' enables it.

In the next post we will try to update this field in the SQL database and (somehow!) get that update pushed to the handheld.

Managing BlackBerry Configuration Settings Remotely - Part One

Much of the configuration of BlackBerry devices can be controlled through IT Policy options, especially those having to do with security. However there are many more configuration options available on the device that don't have IT Policy correlates. In these cases it can be difficult to control the user experience or to enforce standard organizational settings.

There are a few options to automate the configuration of these device settings:

1) Configure a BlackBerry with your custom settings, then back it up to a .IPD file. For each new BlackBerry deployed, restore the .IPD file before activation.

2) Configure a BlackBerry with your custom settings, then assign it to the new user's account. Then activate a new BlackBerry against the same account, which will inherit the device configuration settings from the old device.

These options can set an initial standard, however they cannot be changed remotely or locked down and enforced over time. The BlackBerry device user will have the ability to modify these settings and break the standard.

The fact that option #2 above exists tells us that the device configuration settings are stored within the BES environment, as that is how swapping in brand new device for a user can inherit all the old device's settings. Unfortunately the location of these settings is not easily accessible to the BES Administrator, as this function was really only designed for backup and restoration purposes.

However... if we can figure out where these settings are stored perhaps we can modify them somehow from the back end, and push them down to the device. We know that most configuration information is stored in the central SQL database, so this is where the configuration must be stored as well.

Peeking around my test environment I discovered a table in the database called SyncBackupRestore. It appears that much of the device's configuration settings are stored in this table. In the next post we will dig deeper into this table to discover what it holds and how perhaps we can modify it and push the changes to a remote BlackBerry.

BlackBerry Lookup - The Details (Updated for 4.1 SP4)

After doing some troubleshooting on lookups the other day I discovered that there are now some additional fields that are returned via lookup, that were not there back in the 4.0 days when I posted my first post on this over a year ago.

Specifically, the Home Address fields have now been added, which is nice. Here is a lookup result that includes as the data the Notes Person document field names that the lookup pulls from for reference:


Please note two missing items from this lookup:

1) PIN is missing due to the test Person document not having a BlackBerry device

2) The "Fax: OfficeFAXPhoneNumber" lookup result is missing, though populated in the Person document. This is due to a known issue introduced in SP4 MR1 and scheduled to be fixed in SP5.


Now, onto something interesting that I wasn't aware of before. You are able to customize the lookup results to a certain extent, by using the 4 "User Defined x" fields present in the "Edit PIM Sync Global Field Mappings" command from the BES Manager. [I didn't think this was an option as I always thought that the Global Field Mappings had to do with Address Book synchronization, I never thought they would impact the lookup results.]

Simply choose an unmapped field, drop down the Device Field column field and choose one of the User Defined fields to assign it to:


In my test environment I have assigned the following custom fields:

User Defined 1 --> Assistant
User Defined 1 --> Manager
User Defined 1 --> Spouse
User Defined 1 --> Children

When I do this, I get the following lookup result:

I see the Assistant, Manager, and Spouse fields, but not the Children field I mapped. Perhaps there is an issue with the "User Defined 4" field not working correctly for lookups. Let me change the mapping of this field to "Department" and see what happens:

Interestingly, the Department field does come over. So perhaps it is not an issue with the "User Defined 4" field , but with the Children field not working correctly.

On a final note, the Comments field on the Person document can be multiple lines and all lines will show up on the BlackBerry lookup. So instead of the "User1:" etc mapping above you can create freeform text lines in the Comments field if you need them on the BlackBerry lookup. For example, I added two additional lines to the Comments field in the Person document, and here they are on the lookup:

I'm not sure what the limit on the number of lines or characters are for the free form text Comments field. I was able to put 60 lines of information in that field and they were all returned to the BlackBerry on a lookup, so there is quite a bit of information that could be stored here.

Well I hope this helps you customize your lookups so that you can give your users better information directly on their device. Now the greater problem is how to get all this good information from other sources, such as the HR database, into the Person documents in names.nsf in the first place. (Luckily for me that heavy lifting job was already done by our development team for another project.) Good luck!

Lookup Strangeness -> "Reset PIM Sync Global Field Mapping" is the Fix

I recently discovered that our BlackBerry lookups were returning some bad information. Our addressing information was coming up wrong, in that the Home Address header was missing, replaced by a second Work Address header. Also, the information in the home address fields was limited just to city and country.

Here is what we were seeing (fields filled with dummy info for testing):


The last address is actually the two home address fields city and country, but labeled as a second Work Address. I had never seen this before, at least this was working properly a few months ago. We have a secondary BES infrastructure in the EU, totally separate, but alas this was occuring there too, which pointed to a Domino problem.

I assumed this must be due to some corruption or weird customization in the names.nsf, where the BES pulls this information from. However copying and pasting my person document into my test BES environment, then doing a lookup from a test device, returned all the proper information! Puzzling...

It is difficult to explain this issue to RIM support, and they also could not recreate the solution in their environment. On a whim, I decided to investigate the "Reset PIM Sync Global Field Mapping" command. After testing this in the test environment and determining that this command would not reset user's custom address book mappings, I ran it. Guess what? All lookup problems solved:

The Home Address is now properly labeled, and although blanked out above, the fields for street address, state, and zip code now appear along with city and country.

This solution points to the address book mappings being off kilter within the BES SQL database, but how did that happen? We don't touch them. And since both of our completely separate BES environments experienced the issue, I must believe that a particular upgrade path (we keep versions the same in US and EU) caused this mis-mapping.

Easy solution to a strange issue!

p.s. I discovered some other interesting lookup related information during this search that I will discuss in an upcoming post.

How to Recover* from a Total Loss of BESMgmt SQL database (* with caveats...)

The BESMgmt SQL database is the heart of the Blackberry Enterprise environment. Without it, BES servers will continue to deliver mail but no meaningful changes can be made, or BES services restarted, until it is back up.

The only piece of the BES environment that is out of my control is the database, which is hosted and managed by our SQL team. Although I have full confidence in their replication, backup, and recovery plans, I still need to plan for the (hopefully very remote) possibility that the SQL db might go bye-bye someday.

Until now, this worst case scenario resulted in an "oh well, everybody will have to reactivate their devices" DR plan. I dread the day I have to say this to the CIO!

Recently, however I got wind of the idea of using the nbesmigration.exe utility for DR purposes. This utility was originally designed to populate user accounts in the 4.0 SQL database from a version 2.2 profiles database. Since the 4.x environment still maintains a BlackBerryProfiles notes database on each server, the tool could theoretically be used to re-create the user accounts from these profile entries into a brand new SQL database.

While running the tool in testing, I noticed the following text as part of the built-in help:

BlackBerry Enterprise Server(c) for Lotus Domino(c) Migration Tool, Version 4.1
Copyright (c) Research In Motion, Ltd. 2004, 2006. All rights reserved.
Modification date: May 8 2007

Usage: NBESMigration.exe [options]

This tool can be used to migrate entries from BESD 2.2.x Profiles database to BESMgmt database (UserConfig table). For migration, create the database first using CreateDB, create a server entry in ServerConfig, and run this tool from your BESD server's Domino directory.

This tool can also used for disaster recovery or fixup purposes on already installed BESD server (4.0 and up). Shut down the BES, and run this tool from your BESD server's Domino directory. The tool will check all user docs in Profiles DB, create entries, if needed, in BESMgmt DB (UserConfig table), and it will link entries using SQLID field.

From the text it appears that this tool does support a limited form of DR. Good news! So to test, I did the following in my test environment:

1. Detach the BES SQL db and drop all connections.

2. Stop BES server task and services.

3. Run the BlackBerry Congfiguration Utility and choose the Change Database option.

4. Enter a different database name, thus creating a brand new database.

5. Re-enter CAL and SRP information, but do not restart services.

6. From the tools subdirectory of the BES 4.1 installation share, copy the nbesmigration.exe executable to the c:\lotus\domino directory

7. Run the following command nbesmigration -d: "[SQL Server name]" "[SQL DB name]" -u: [SQL login] [password]. In my case this command line looked like this:

nbesmigration -d: "SQLTEST01" "BESMgmtTest" -u: besdbowner besdbpassword

The tool indicated that it found my one test account and migrated it successfully.

8. Restart the BES server task and services.

The server fired up and I was able to immediately send and receive messages, and do lookups from my device! So this is a great way to recover basic functionality in the case of a total loss of the SQL db.

Keep in mind that you need some information documented and handy to provide to the configuration wizard:

• CAL licenses
• SRP ID's
• SRP Authorization keys

Now, for the caveats:

• IT Policies are reset to defaults (!)
  
• PIM Sync Mappings are reset to defaults
• Email Settings are reset to defaults
• Probably a bunch of other stuff is gone or reset to defaults

Although some of the other stuff is fixable, I was unable to successfully apply the default IT policy or a newly created IT policy in the new enviroment, even when it was named the same as the old IT policy. It always came back with an error. On the other hand, when I switched DB's back to the original, the original IT policy was able to be re-applied successfully.

The upshot is that this is a last-last-resort option to get mail flowing to the devices without having everyone reactivate. If you can later restore the original SQL DB and switch back to it, then IT Policies, PIM Sync settings, etc, can be restored back to their original good state.

If you can never get the original DB back, however, then you might have to move forward and find some creative solutions to the problems above. If the solution does eventually require handheld reactivation, well then at least you can control and stage it on your terms, without having hundreds of people screaming at you to get their BlackBerry email fixed!

Last Handheld Contact Times - Bogus?

This is a useful field to know when the last time the server talked to the handheld... until I started to suspect this information, as there were contact times very recently with handhelds that I knew were disabled.

To test this out I turned off my BlackBerry and restarted the BES task. The last handheld contact time updated to the current time, with the device turned off in front of me! See the below screenshot, showing last contact time of 1/15/2008 10:16:51AM, when the device was turned off at about 10:10 AM.


To see why this is updating, I checked the MAGT logs for activity on my account during this time:

[40000] (01/15 10:16:51.733):{0xB04} {Test User/Domain} Sending PIM Transaction to Sync Server, Tag 225 (config request)
[40000] (01/15 10:16:51.733):{0x1F9C} [BIPP] Send data, Tag=225
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received status DELIVERED, Tag=225
[40392] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SRP: TID=225, type PIMSYNC returned DELIVERED
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received datagram, Tag=436
[40076] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SendStatusToWirelessNetworkUsingSRP SendQ, TID=436
[40400] (01/15 10:16:51.748):{0xB04} {Test User/Domain} Received datagram with content type agent_sync, TID=436, for user Test User/Domain
[40000] (01/15 10:16:51.748):{0x1F9C} [BIPP] Send status DATA_ACCEPTED, Tag=436

The only activity that occured was an internal communication to the Sync server, and what looks like a request to perform a handheld agent sync. Note that both transactions appear to show successful connections with the device, however these are only internal communications between components on the BES server itself. Perhaps the time of successful internal communication is being reported?

In any case, it appears that this is quite a bogus field which provides no useful real world information.