BlackBerry Lookup - The Details (Updated for 4.1 SP4)

After doing some troubleshooting on lookups the other day I discovered that there are now some additional fields that are returned via lookup, that were not there back in the 4.0 days when I posted my first post on this over a year ago.

Specifically, the Home Address fields have now been added, which is nice. Here is a lookup result that includes as the data the Notes Person document field names that the lookup pulls from for reference:


Please note two missing items from this lookup:

1) PIN is missing due to the test Person document not having a BlackBerry device

2) The "Fax: OfficeFAXPhoneNumber" lookup result is missing, though populated in the Person document. This is due to a known issue introduced in SP4 MR1 and scheduled to be fixed in SP5.


Now, onto something interesting that I wasn't aware of before. You are able to customize the lookup results to a certain extent, by using the 4 "User Defined x" fields present in the "Edit PIM Sync Global Field Mappings" command from the BES Manager. [I didn't think this was an option as I always thought that the Global Field Mappings had to do with Address Book synchronization, I never thought they would impact the lookup results.]

Simply choose an unmapped field, drop down the Device Field column field and choose one of the User Defined fields to assign it to:


In my test environment I have assigned the following custom fields:

User Defined 1 --> Assistant
User Defined 1 --> Manager
User Defined 1 --> Spouse
User Defined 1 --> Children

When I do this, I get the following lookup result:

I see the Assistant, Manager, and Spouse fields, but not the Children field I mapped. Perhaps there is an issue with the "User Defined 4" field not working correctly for lookups. Let me change the mapping of this field to "Department" and see what happens:

Interestingly, the Department field does come over. So perhaps it is not an issue with the "User Defined 4" field , but with the Children field not working correctly.

On a final note, the Comments field on the Person document can be multiple lines and all lines will show up on the BlackBerry lookup. So instead of the "User1:" etc mapping above you can create freeform text lines in the Comments field if you need them on the BlackBerry lookup. For example, I added two additional lines to the Comments field in the Person document, and here they are on the lookup:

I'm not sure what the limit on the number of lines or characters are for the free form text Comments field. I was able to put 60 lines of information in that field and they were all returned to the BlackBerry on a lookup, so there is quite a bit of information that could be stored here.

Well I hope this helps you customize your lookups so that you can give your users better information directly on their device. Now the greater problem is how to get all this good information from other sources, such as the HR database, into the Person documents in names.nsf in the first place. (Luckily for me that heavy lifting job was already done by our development team for another project.) Good luck!

Lookup Strangeness -> "Reset PIM Sync Global Field Mapping" is the Fix

I recently discovered that our BlackBerry lookups were returning some bad information. Our addressing information was coming up wrong, in that the Home Address header was missing, replaced by a second Work Address header. Also, the information in the home address fields was limited just to city and country.

Here is what we were seeing (fields filled with dummy info for testing):


The last address is actually the two home address fields city and country, but labeled as a second Work Address. I had never seen this before, at least this was working properly a few months ago. We have a secondary BES infrastructure in the EU, totally separate, but alas this was occuring there too, which pointed to a Domino problem.

I assumed this must be due to some corruption or weird customization in the names.nsf, where the BES pulls this information from. However copying and pasting my person document into my test BES environment, then doing a lookup from a test device, returned all the proper information! Puzzling...

It is difficult to explain this issue to RIM support, and they also could not recreate the solution in their environment. On a whim, I decided to investigate the "Reset PIM Sync Global Field Mapping" command. After testing this in the test environment and determining that this command would not reset user's custom address book mappings, I ran it. Guess what? All lookup problems solved:

The Home Address is now properly labeled, and although blanked out above, the fields for street address, state, and zip code now appear along with city and country.

This solution points to the address book mappings being off kilter within the BES SQL database, but how did that happen? We don't touch them. And since both of our completely separate BES environments experienced the issue, I must believe that a particular upgrade path (we keep versions the same in US and EU) caused this mis-mapping.

Easy solution to a strange issue!

p.s. I discovered some other interesting lookup related information during this search that I will discuss in an upcoming post.

How to Recover* from a Total Loss of BESMgmt SQL database (* with caveats...)

The BESMgmt SQL database is the heart of the Blackberry Enterprise environment. Without it, BES servers will continue to deliver mail but no meaningful changes can be made, or BES services restarted, until it is back up.

The only piece of the BES environment that is out of my control is the database, which is hosted and managed by our SQL team. Although I have full confidence in their replication, backup, and recovery plans, I still need to plan for the (hopefully very remote) possibility that the SQL db might go bye-bye someday.

Until now, this worst case scenario resulted in an "oh well, everybody will have to reactivate their devices" DR plan. I dread the day I have to say this to the CIO!

Recently, however I got wind of the idea of using the nbesmigration.exe utility for DR purposes. This utility was originally designed to populate user accounts in the 4.0 SQL database from a version 2.2 profiles database. Since the 4.x environment still maintains a BlackBerryProfiles notes database on each server, the tool could theoretically be used to re-create the user accounts from these profile entries into a brand new SQL database.

While running the tool in testing, I noticed the following text as part of the built-in help:

BlackBerry Enterprise Server(c) for Lotus Domino(c) Migration Tool, Version 4.1
Copyright (c) Research In Motion, Ltd. 2004, 2006. All rights reserved.
Modification date: May 8 2007

Usage: NBESMigration.exe [options]

This tool can be used to migrate entries from BESD 2.2.x Profiles database to BESMgmt database (UserConfig table). For migration, create the database first using CreateDB, create a server entry in ServerConfig, and run this tool from your BESD server's Domino directory.

This tool can also used for disaster recovery or fixup purposes on already installed BESD server (4.0 and up). Shut down the BES, and run this tool from your BESD server's Domino directory. The tool will check all user docs in Profiles DB, create entries, if needed, in BESMgmt DB (UserConfig table), and it will link entries using SQLID field.

From the text it appears that this tool does support a limited form of DR. Good news! So to test, I did the following in my test environment:

1. Detach the BES SQL db and drop all connections.

2. Stop BES server task and services.

3. Run the BlackBerry Congfiguration Utility and choose the Change Database option.

4. Enter a different database name, thus creating a brand new database.

5. Re-enter CAL and SRP information, but do not restart services.

6. From the tools subdirectory of the BES 4.1 installation share, copy the nbesmigration.exe executable to the c:\lotus\domino directory

7. Run the following command nbesmigration -d: "[SQL Server name]" "[SQL DB name]" -u: [SQL login] [password]. In my case this command line looked like this:

nbesmigration -d: "SQLTEST01" "BESMgmtTest" -u: besdbowner besdbpassword

The tool indicated that it found my one test account and migrated it successfully.

8. Restart the BES server task and services.

The server fired up and I was able to immediately send and receive messages, and do lookups from my device! So this is a great way to recover basic functionality in the case of a total loss of the SQL db.

Keep in mind that you need some information documented and handy to provide to the configuration wizard:

• CAL licenses
• SRP ID's
• SRP Authorization keys

Now, for the caveats:

• IT Policies are reset to defaults (!)
  
• PIM Sync Mappings are reset to defaults
• Email Settings are reset to defaults
• Probably a bunch of other stuff is gone or reset to defaults

Although some of the other stuff is fixable, I was unable to successfully apply the default IT policy or a newly created IT policy in the new enviroment, even when it was named the same as the old IT policy. It always came back with an error. On the other hand, when I switched DB's back to the original, the original IT policy was able to be re-applied successfully.

The upshot is that this is a last-last-resort option to get mail flowing to the devices without having everyone reactivate. If you can later restore the original SQL DB and switch back to it, then IT Policies, PIM Sync settings, etc, can be restored back to their original good state.

If you can never get the original DB back, however, then you might have to move forward and find some creative solutions to the problems above. If the solution does eventually require handheld reactivation, well then at least you can control and stage it on your terms, without having hundreds of people screaming at you to get their BlackBerry email fixed!

Last Handheld Contact Times - Bogus?

This is a useful field to know when the last time the server talked to the handheld... until I started to suspect this information, as there were contact times very recently with handhelds that I knew were disabled.

To test this out I turned off my BlackBerry and restarted the BES task. The last handheld contact time updated to the current time, with the device turned off in front of me! See the below screenshot, showing last contact time of 1/15/2008 10:16:51AM, when the device was turned off at about 10:10 AM.


To see why this is updating, I checked the MAGT logs for activity on my account during this time:

[40000] (01/15 10:16:51.733):{0xB04} {Test User/Domain} Sending PIM Transaction to Sync Server, Tag 225 (config request)
[40000] (01/15 10:16:51.733):{0x1F9C} [BIPP] Send data, Tag=225
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received status DELIVERED, Tag=225
[40392] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SRP: TID=225, type PIMSYNC returned DELIVERED
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received datagram, Tag=436
[40076] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SendStatusToWirelessNetworkUsingSRP SendQ, TID=436
[40400] (01/15 10:16:51.748):{0xB04} {Test User/Domain} Received datagram with content type agent_sync, TID=436, for user Test User/Domain
[40000] (01/15 10:16:51.748):{0x1F9C} [BIPP] Send status DATA_ACCEPTED, Tag=436

The only activity that occured was an internal communication to the Sync server, and what looks like a request to perform a handheld agent sync. Note that both transactions appear to show successful connections with the device, however these are only internal communications between components on the BES server itself. Perhaps the time of successful internal communication is being reported?

In any case, it appears that this is quite a bogus field which provides no useful real world information.

MDS What?

MDS stands for Mobile Data Service, and it is a service that allows you to access other data sources (mostly Internet / Intranet browsing) via your BlackBerry.

In BES 4.1, however, MDS has turned into a many-headed hydra. What we know of as "BlackBerry Mobile Data Service" in 4.0 is now inexplicably called the "BlackBerry MDS Connection Service" in 4.1. Meanwhile, there are a whole bunch of new MDS related services:

•BlackBerry MDS Application Integration Service
•BlackBerry MDS Data Optimization Service
•BlackBerry MDS Provisioning Service
•BlackBerry MDS Administrative and Management Service
•BlackBerry® MDS Studio Application Repository

But wait, there's more! The new MDS stuff also requires a brand new SQL db running on the same SQL server as the BESMgmt database, which can results in permission issues when creating without the proper authority.

All of the above services & database are related to the new 4.1 software deployment environment installed when you choose the upgrade option called:

"BlackBerry Enterprise Server with MDS Services and Components"

But really, this whole new "MDS" software deployment infrastructure is not needed at the initial upgrade from 4.0 to 4.1. To avoid complexity, you can leave this whole new set of services out of the upgrade. Later, you can install the new MDS stuff on a separate server if you like, as it was made to be modular and have one separate MDS instance serve many BES servers.

The confusing part is that if you run the upgrade, you get these two choices:


If I do not know *exactly* what this means, I will by default choose the second option, "BlackBerry Enterprise Server with MDS Services and Components", because I want to keep my MDS service from 4.0, right?

Wrong... choosing the first option will give you the same 4.0 MDS (renamed MDS Connection Service) while avoiding the complexity of installing the new whiz-bang MDS software deployment stuff.

In a nutshell: if you want to greatly simplify your 4.0 -> 4.1 upgrade, opt for the first install method selected in the picture above. You will lose no MDS functionality from the 4.0 perspective, and can add on the new stuff later when you are ready and comfortable with 4.1.

Malformed Message Crashes BES 4.1 SP4

The idea that a malformed message will crash a BES server is nothing new - service packs have taken care of these issues many times in the past. I apparently discovered another one, as one of my servers crashed twice just after midnight last night.

Fortunately Domino restarted itself and was back up and operational in minutes (thanks transaction logging!). After the second crash, however, it did not crash again. Usually the BES will keep trying to re-read the malformed message and crash over and over until you figure out the message and delete it from the user's mailfile, but not this time.

From the logs I see the attempts to read which resulted in crashes:

[40000] (11/01 00:00:44.858):{0x19B0} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:00:44.905):{0x19B0} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

But on the third attempt I see this:

[40000] (11/01 00:08:10.515):{0x1640} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM
[20039] (11/01 00:08:10.530):{0x1640} {User} Already attempted to open NID=3DAF2 for user User: Message has been quarantined, skipping now

Nice job RIM! This quarantining feature allowed me to stay peacefully asleep instead of having to get up and hunt down the offending message.

BTW, the message in question was a digest from a mailing list which included a BinHex encoded MIME part in the body of the message:

--B_3276667131_13573
Content-type: application/mac-binhex40; name="[Filename].doc"
Content-disposition: attachment;
filename="[Filename].doc"

(This file must be converted with BinHex 4.0)
:(8K[G#p1Eh3J9A"NBA4P)#dJ5R9XH5!R-$FZC'pM!&Fi3Nj08eG%!*!%UJ#3"GN
Jd-m4i+'a'Z%!N"!q!!-!r[m*!!B!N!X"!*!$8!#3#"!!!&)!N!-"!*!$r[q3!`#
3"%m!N!2rN2rrN,(XTF%!Kf%*"!!!q"+r!*!&!4%!!3!"!!B!!28L!!!1!'TLDQ+
`Zl#l!*!5#33@!#3d!!$Df3%!fYN"!28F!*!Hrrm2!*!*rrm2!*!*rrm2!*!4L!#


I am pretty sure this is the part that utterly confused the BES since:

a) it was in the body

and

b) was a binhex part, which I have seen trouble with in prior BES versions even when it was properly encoded.

Looking through the release notes for 4.1 SP4 MR2, I see the following:

*SDR 135729 In BlackBerry Enterprise Server Version 4.1 SP4, if a message contained truncated or incorrectly encoded data, the BlackBerry Enterprise Server might have stopped responding. In BlackBerry Enterprise Server Version 4.1 SP4 MR2 and later, this issue is resolved.

I love to see this - it means I don't have to report the issue to RIM, someone else already has! Also I can tell my manager when he asks about the crashes that it is already fixed in the next maintenance release, which we will now plan on deploying.

Decoding the BlackBerry State Database

So we all know that state database correlates the messages in the mailfile with the messages on the BlackBerry. The way it does this is by creating a separate entry in the state database for each email the BES sees in the mailfile. Each of these entries has exactly the same UNID as the original email in the state database.

Now I am trying to decode the MessageState field in the state database entries. Searching through my own state database, I find all of the following various "state codes" under the MessageState field:

0
1
2
3
4
5
7
8
9
10
13
14
17

That's alot of "states" a message can be in!

I have started testing and have determined the first couple:

2: Email has been queued / sent to wireless network but not received by device yet
3: Email not redirected to handheld (redirection disabled)
4: Email has been delivered to device

It will be tough to figure out the rest, though! I will update this doc as I discover them...