Last Handheld Contact Times - Bogus?

This is a useful field to know when the last time the server talked to the handheld... until I started to suspect this information, as there were contact times very recently with handhelds that I knew were disabled.

To test this out I turned off my BlackBerry and restarted the BES task. The last handheld contact time updated to the current time, with the device turned off in front of me! See the below screenshot, showing last contact time of 1/15/2008 10:16:51AM, when the device was turned off at about 10:10 AM.


To see why this is updating, I checked the MAGT logs for activity on my account during this time:

[40000] (01/15 10:16:51.733):{0xB04} {Test User/Domain} Sending PIM Transaction to Sync Server, Tag 225 (config request)
[40000] (01/15 10:16:51.733):{0x1F9C} [BIPP] Send data, Tag=225
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received status DELIVERED, Tag=225
[40392] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SRP: TID=225, type PIMSYNC returned DELIVERED
[40000] (01/15 10:16:51.748):{0x19E0} [BIPP] Received datagram, Tag=436
[40076] (01/15 10:16:51.748):{0xB04} {Test User/Domain} SendStatusToWirelessNetworkUsingSRP SendQ, TID=436
[40400] (01/15 10:16:51.748):{0xB04} {Test User/Domain} Received datagram with content type agent_sync, TID=436, for user Test User/Domain
[40000] (01/15 10:16:51.748):{0x1F9C} [BIPP] Send status DATA_ACCEPTED, Tag=436

The only activity that occured was an internal communication to the Sync server, and what looks like a request to perform a handheld agent sync. Note that both transactions appear to show successful connections with the device, however these are only internal communications between components on the BES server itself. Perhaps the time of successful internal communication is being reported?

In any case, it appears that this is quite a bogus field which provides no useful real world information.

MDS What?

MDS stands for Mobile Data Service, and it is a service that allows you to access other data sources (mostly Internet / Intranet browsing) via your BlackBerry.

In BES 4.1, however, MDS has turned into a many-headed hydra. What we know of as "BlackBerry Mobile Data Service" in 4.0 is now inexplicably called the "BlackBerry MDS Connection Service" in 4.1. Meanwhile, there are a whole bunch of new MDS related services:

•BlackBerry MDS Application Integration Service
•BlackBerry MDS Data Optimization Service
•BlackBerry MDS Provisioning Service
•BlackBerry MDS Administrative and Management Service
•BlackBerry® MDS Studio Application Repository

But wait, there's more! The new MDS stuff also requires a brand new SQL db running on the same SQL server as the BESMgmt database, which can results in permission issues when creating without the proper authority.

All of the above services & database are related to the new 4.1 software deployment environment installed when you choose the upgrade option called:

"BlackBerry Enterprise Server with MDS Services and Components"

But really, this whole new "MDS" software deployment infrastructure is not needed at the initial upgrade from 4.0 to 4.1. To avoid complexity, you can leave this whole new set of services out of the upgrade. Later, you can install the new MDS stuff on a separate server if you like, as it was made to be modular and have one separate MDS instance serve many BES servers.

The confusing part is that if you run the upgrade, you get these two choices:


If I do not know *exactly* what this means, I will by default choose the second option, "BlackBerry Enterprise Server with MDS Services and Components", because I want to keep my MDS service from 4.0, right?

Wrong... choosing the first option will give you the same 4.0 MDS (renamed MDS Connection Service) while avoiding the complexity of installing the new whiz-bang MDS software deployment stuff.

In a nutshell: if you want to greatly simplify your 4.0 -> 4.1 upgrade, opt for the first install method selected in the picture above. You will lose no MDS functionality from the 4.0 perspective, and can add on the new stuff later when you are ready and comfortable with 4.1.

Malformed Message Crashes BES 4.1 SP4

The idea that a malformed message will crash a BES server is nothing new - service packs have taken care of these issues many times in the past. I apparently discovered another one, as one of my servers crashed twice just after midnight last night.

Fortunately Domino restarted itself and was back up and operational in minutes (thanks transaction logging!). After the second crash, however, it did not crash again. Usually the BES will keep trying to re-read the malformed message and crash over and over until you figure out the message and delete it from the user's mailfile, but not this time.

From the logs I see the attempts to read which resulted in crashes:

[40000] (11/01 00:00:44.858):{0x19B0} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:00:44.905):{0x19B0} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], ModifiedByName detected change
[40000] (11/01 00:05:02.452):{0x1830} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM

..[CRASH HERE!]..

But on the third attempt I see this:

[40000] (11/01 00:08:10.515):{0x1640} {User} [Mailfile], fetching modified documents since 11/01/2007 12:00:43 AM
[20039] (11/01 00:08:10.530):{0x1640} {User} Already attempted to open NID=3DAF2 for user User: Message has been quarantined, skipping now

Nice job RIM! This quarantining feature allowed me to stay peacefully asleep instead of having to get up and hunt down the offending message.

BTW, the message in question was a digest from a mailing list which included a BinHex encoded MIME part in the body of the message:

--B_3276667131_13573
Content-type: application/mac-binhex40; name="[Filename].doc"
Content-disposition: attachment;
filename="[Filename].doc"

(This file must be converted with BinHex 4.0)
:(8K[G#p1Eh3J9A"NBA4P)#dJ5R9XH5!R-$FZC'pM!&Fi3Nj08eG%!*!%UJ#3"GN
Jd-m4i+'a'Z%!N"!q!!-!r[m*!!B!N!X"!*!$8!#3#"!!!&)!N!-"!*!$r[q3!`#
3"%m!N!2rN2rrN,(XTF%!Kf%*"!!!q"+r!*!&!4%!!3!"!!B!!28L!!!1!'TLDQ+
`Zl#l!*!5#33@!#3d!!$Df3%!fYN"!28F!*!Hrrm2!*!*rrm2!*!*rrm2!*!4L!#


I am pretty sure this is the part that utterly confused the BES since:

a) it was in the body

and

b) was a binhex part, which I have seen trouble with in prior BES versions even when it was properly encoded.

Looking through the release notes for 4.1 SP4 MR2, I see the following:

*SDR 135729 In BlackBerry Enterprise Server Version 4.1 SP4, if a message contained truncated or incorrectly encoded data, the BlackBerry Enterprise Server might have stopped responding. In BlackBerry Enterprise Server Version 4.1 SP4 MR2 and later, this issue is resolved.

I love to see this - it means I don't have to report the issue to RIM, someone else already has! Also I can tell my manager when he asks about the crashes that it is already fixed in the next maintenance release, which we will now plan on deploying.

Decoding the BlackBerry State Database

So we all know that state database correlates the messages in the mailfile with the messages on the BlackBerry. The way it does this is by creating a separate entry in the state database for each email the BES sees in the mailfile. Each of these entries has exactly the same UNID as the original email in the state database.

Now I am trying to decode the MessageState field in the state database entries. Searching through my own state database, I find all of the following various "state codes" under the MessageState field:

0
1
2
3
4
5
7
8
9
10
13
14
17

That's alot of "states" a message can be in!

I have started testing and have determined the first couple:

2: Email has been queued / sent to wireless network but not received by device yet
3: Email not redirected to handheld (redirection disabled)
4: Email has been delivered to device

It will be tough to figure out the rest, though! I will update this doc as I discover them...

Remove IT Policy - New Feature of 4.1 SP4 / OS 4.2.2

We now have an easy way to remove the applied IT Policy from a handheld, which was previously locked to the device even after you wiped it.

If you have 4.1 SP4 and a device with OS 4.2.2 or greater, here is how to remove the IT policy and set the device to a true factory default state:

1) Create a new IT policy (or modify an existing policy) and set the Remote Wipe Reset to Factory Defaults setting in the Security Policy group to True



2) Assign this IT policy to a particular user account
3) Send the command "Erase Data and Disable Handheld"



4) Verify receipt of command by device under "IT Policy Status"

Update on Comcast's Man in the Middle Attacks on Lotus Notes

For those of you Notes client users suffering from Comcast's "filtering", there is a great post by my colleague Kevin Kanarski who has details of packet captures that result from sending an email with a 6MB attachment from a Notes client using the Internet connection.

These captures, from both ends, clearly show that Comcast is imitating both the client and server in sending RST [reset] packets to the other end of the connection. Neither the client nor the server generated any RST packets, so this is definitely shady behavior by Comcast.

Tuning the "More" Cache

We all know the pesky little "More" command that lets you download successively more and more of your email, beyond the first 4K delivered to the BlackBerry.

Well on the server side, there is a cache that holds this information, and it is called the "More cache". [Wow these names really need a little obfuscation, they make too much sense to exist in the PC world]

Anyhoo, you can tune the amount of cache that the BES allocates to this function, and by default it gives it 10MB. Well at this level I was filling up the cache in about 6 hours from server startup, and only realizing a maximum of 84% hit rate:

[45009] (09/14 02:35:58.078):{0x13BC} More cache hit rate: 0.0%, requests: 0, adds: 0, size: 0.0/10.0 Mb
...
[45009] (09/14 08:33:32.778):{0x13BC} More cache hit rate: 69.0%, requests: 126, adds: 2270, size: 5.7/10.0 Mb
...
[45009] (09/14 09:48:32.294):{0x13BC} More cache hit rate: 75.7%, requests: 214, adds: 5031, size: 10.0/10.0 Mb
...
[45009] (09/14 23:48:32.111):{0x13BC} More cache hit rate: 84.4%, requests: 784, adds: 29120, size: 10.0/10.0 Mb


If the remaining text is not found in the More cache, then the BES server needs to open the mailfile to pull it in, which is expensive in terms of network I/O and Domino server I/O. [Not hugely, of course, but I like to squeeze as much performance out of my systems as possible] If it can find it in the cache, here is what you will see in your MAGT log:

[40231] (09/14 01:11:01.290):{0x608} {XXXX} Original message (RID=-576019096) retrieved from More cache. Not opening the mail file.

That is a message that I like to see!

So I decided to double the default of 10MB to 20MB by adding the following DWORD registry key, and setting it to "20" decimal:

HKLM\Software\Research In Motion\BlackBerry Enterprise Server\Agents\MoreCacheSize


After a restart of the server (not required) here is what I now see:

[45009] (09/20 13:50:51.888):{0x14E0} More cache hit rate: 89.5%, requests: 3004, adds: 127078, size: 20.0/20.0 Mb

So fully loaded, it is running ~90% cache hit rate now, which is not a huge gain but it is something. RIM considers >90% hit rates to be good, but I haven't decided how much more I want to push this setting as it takes away memory from other processes for this task.