Comcast Blocking Lotus Notes Attachments over 2MB

This is not related to Domino BES in any way, but thought I would share. It appears that Comcast (at least in the Chicago area) has recently installed a blocking system in order to combat P2P traffic, which has the inadvertent effect of blocking Lotus Notes attachment uploads over 2MB.

Using Lotus Notes with the internet passthrough, via a home Comcast cable IP connection, and trying to send any message over 2MB will result in a bunch of IP reset (RSET) commands sent by the Comcast blocking system which blocks the message from being sent.

We have received 2 calls in the last week regarding this issue, and 4 other tech people with Comcast cable modems have confirmed that sending attachments over 2MB no longer works. The error received is the following:


"Remote system no longer responding: [Server name] mail.box"


If you try to send a message just under 2MB you might see the following:


"Mail was successfully submitted for delivery but a copy has not yet been saved in your mail file due to server not responding"

Has anyone else received complaints from home / laptop Notes users about this issue?

UPDATE: This appears to be confirmed by a posting at www.dslreports.com which discusses Comcast's use of a filtering product called "Sandvine":

http://www.dslreports.com/forum/r18323368-Comcast-is-using-Sandvine-to-manage-P2P-Connections

BlackBerry Date/Time Source Explained

You have three options of how to get the Date/Time synchronized automatically on the BlackBerry:

1. Set it to BlackBerry (the default), which gathers the information from the BlackBerry network, i.e. the RIM NOC.

There is some confusion about this, let's clear it up right now. Setting this to BlackBerry does not mean you have to set the date/time yourself on the BlackBerry. It does not mean that it gets the time from Desktop Manager when you cradle / sync. (that is so 2005 anyways, it's all wireless now baby!) It means that you get the date/time directly from RIM over the air, that's it. Here is the debug log lines from the device when you click Update Time:

guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:08 2007 severity:5 type:2 app:net.rim.timesync data:SynR
guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:08 2007 severity:0 type:2 app:net.rim.timesync data:Send
guid:0x1295B4AADE149AFC time: Thu Aug 02 15:12:10 2007 severity:0 type:2 app:net.rim.timesync data:Recv

2. Set it to Network, which gathers the information from the wireless carrier network, whether AT&T, T-Mobile, Verizon, Rogers, O2, Vodafone, etc. Here is the debug log lines from the device when you do this:

guid:0x1295B4AADE149AFC time: Thu Aug 02 15:15:55 2007 severity:5 type:2 app:net.rim.timesync data:SynR

Interestingly, when you click Update Time with the Network setting, it just copies the date/time recorded under "Network Time" and "Network Date", instead of directly querying the carrier network. I assume this "Network Time" is regularly synced through some background mobile radio process.

3. Set if to Off, which does not gather any info. You set it and maintain it yourself.

Please note that, as I explained in my prior post, none of these settings will update the time zone you are in to the local time zone when you travel. None of them. Stop trying to change from Network to BlackBerry and back and clicking "Update Time" over and over again. It just won't do it.

Automatic TimeZone Switching for BlackBerry

The Motorola Q does it, the Treo does it, the Razr does it... and yet NO BlackBerry does it. What am I talking about? The seemingly obvious and basic cellular phone function of automatically switching the time zone for you while you travel.

This does not require internal GPS, as the other phones do not have/need it and yet have gotten their location / time zone directly from the network they attach to for years.

Now I have seen multiple discussions on the forums complaining/questioning about this feature, where "switchers" to BlackBerry just can't understand how to enable this feature on their BlackBerry... it is such an obvious feature that they think they are stupid and don't know how to get it to work, rather than the ridiculous notion that maybe this feature does not exist on the BlackBerry!

Arguments about setting the Date/Time source to Network vs. BlackBerry, the use of GPS, GPS not required, A-GPS vs internal GPS, it is just an endless circular conversation I have seen over and over with little resolution.

There have been justifications floated around that since the BlackBerry is a multifunction device, not just a phone, that the calendar items would be off whenever you traveled to a new time zone, causing mass confusion. For this reason they left it to the user to manually change the time zone.

This makes little sense to me, in that it might cause me mass confusion to have my calendar appointment times off from the local time I am in! In any case, RIM could even leave it to the user by default, but at least put it there as an option to turn on.

My theory on this, using no background information at all, is that RIM simply does not trust the wireless carrier networks to provide an accurate time zone setting from their networks. If one of the carriers passes out bad time zone data and messes up someone's entire calendar then RIM would be blamed. That is just a theory on my part, but that fear does not necessitate leaving out even the *option* of enabling this feature on the device.

So at WES a few months back I came armed with a couple of questions I was going to get answers to... one of them being this issue. The people I talked to - who granted were not the go to handheld development people - did not express any sort of justification for leaving the functionality off. In fact, they expressed the idea that "Yeah, that makes sense, we should look into enabling a feature like that."

There must be a back story I am missing here, there must have been some conversation about this at some conference table at some time in the 5 years since the first BlackBerry phone was released. Can someone enlighten me... please?

Dedicated Attachment Service Gotcha

Early on we decided to separate our attachment service onto it's own dedicated server in order to offload that work from the BES servers. So now all 3 production BES servers point to this one attachment server.

One thing I noticed early on is that the BES servers cannot re-attach to the attachment server automatically. That is, if you restart the attachment server/service, attachment decoding will fail for all BlackBerry users until the associated BES task is restarted on all the associated BES servers.

The connection to the remote attachment server is handled within the BES Domino task, and the first time it connects upon startup, the BES task creates a special ID associated with that server's connection. When the attachment server is restarted, that connection ID no longer exists and the attachment server will reject decode requests from the BES server.

It is a simple fix (restarting BES task), but disruptive and can cause headaches during our monthly maintenance window - I have to always remember to restart the attachment server first before any of the BES servers are restarted.

This issue occurred in the 4.0 environment however it would be interesting to see if it still occurs in the 4.1 environment. When I find out I will post an update, however I haven't seen any SDR's addressing this issue as of yet.

One more thing to watch out for in a complex distributed BES environment!

UPDATE 7/24/07: It appears that this issue was addressed in 4.1 SP4, so no more reboot juggling required!

Thinking of removing your DST software config? Read this...

Thinking that this was no longer a large issue, I nonchalantly removed the DST software config from ~1300 users last week. About a few hours later I get some inquiries from the regional offices to the effect of:

"We had some people call about receiving a 'Permission's restrictions need to be updated, do you want to restart your device now or later' prompt on their handheld. They accepted it and the device restarted, but I had never seen this before, what is it?"

This is the "Oh s***" moment, when you realize that something seemingly innocuous had directly impacted each and every user you modified in bulk.

You could say this was pretty minor, their functionality was fine and all they needed to do was restart the device, but that type of stuff doesn't fly in our environment. Everything that directly impacts the user requires a change control approval process.

Time to move the function "unapply software configuration" from daily administrative task to change-controlled lockdown!

Apparently this happens when you initially followed RIM's recommendation and created a specific software policy for DST which allowed external communications. Removing this software policy reverts it back to some default policy which requires a restart, hence the user prompt.

Oh well, live and learn. (...and share the pain!)

Lotus Domino TimeDate Structure

This structure is used in many places within Domino, such as creation / modification times as well as database and replica ID's. Here is a diagram showing just what nuggets of interesting information are stored in this critical Notes data structure:

Analysis: "Transaction Error: Failure at Service"

Had this error pop up on a handheld today when trying to send from the device, got the dreaded red X. Receiving messages, however, was fine. Usually this boils down to one of a couple of issues:

1) BES server issue
2) Carrier issue
3) Service Book issue

Since the BES was fine and the carrier seemed to be OK for other users, it must have been a Service Book issue. Tried deleting and then undeleting the Desktop [CMIME] service book, didn't work. Tried deleting the Desktop [CMIME] service book, then pushing service books from the BES. Didn't work. Since lookups were also having a problem, I theorized that the odds of 2 service books (lookups are the Desktop [ALP] service book) getting corrupted was pretty small, so time to move onto the next possible culprit.

To take the carrier out of the picture, I hooked the device up to USB so that the messages would be sent using the BlackBerry Router over USB / LAN, totally bypassing the wireless carrier and thus getting that variable out of the picture.

Well I still got the Red X. Hmmm.... but this time it did not specify "Failure at Service". This time the sub-error was "Decryption Failure". Interesting: so now we are getting down to the root of the problem and I can definitely say it is not the carrier or the service books.

So I check the Dispatcher logs, since it is responsible for encryption/decryption to the device, and voila:

[30368] (06/05 10:41:18):{0xFC0} {User} Packet has been delivered to device, Tag=4758107
[40700] (06/05 10:41:44):{0xD8} {User} Receiving packet from device, size=127, TransactionId=1429352292, Tag=259213, content type=ITADMIN, cmd=0x3
[20209] (06/05 10:41:44):{0xD8} {User} DecryptDecompress() failed, Tag=259213, Error=604
[40275] (06/05 10:41:44):{0xD8} {User} Sending transaction error to device for transaction 1429352292, size=46, TransactionId=-966799068, Tag=4758202

So now I know it is an encryption issue, so let's generate a new key:

[40700] (06/05 11:05:13.616):{0x1468} {User} Receiving packet from device, size=225, TransactionId=1429352316, Tag=7143787, content type=OTAKEYGEN, cmd
=0x3
[30222] (06/05 11:05:13.616):{0x1468} {User} MFH: contentType=OTAKEYGEN, sizeOTA=180, sizeOTW=180, TransactionId=1429352316, Tag=7143787
[30308] (06/05 11:05:13.616):{0x1468} [BIPPa] {User} Forwarding data to BES Agent (S45776843), size=218, intTag=1911024, Tag=7143787
[30311] (06/05 11:05:13.616):{0x1438} {User} Forwarding status to relay, intTag=1911024, Tag=7143787, Status=1
[40279] (06/05 11:05:13.616):{0x1438} {User} SubmitToRelaySendQ, Tag=7143787
[30222] (06/05 11:05:13.616):{0x1288} {User} MTH: contentType=OTAKEYGEN, sizeOTA=14, sizeOTW=14, TransactionId=-2070675455, Tag=6675317
[30310] (06/05 11:05:13.616):{0x1288} {User} Forwarding internal data to device, contentType=OTAKEYGEN, routing=S45776843, device=23F7A64F, size=55, cm
d=0x3, ack=0, TransactionId=-2070675455, intTag=7486040, Tag=6675317, Submit=1
[40279] (06/05 11:05:13.616):{0x1288} {User} SubmitToRelaySendQ, Tag=6675317

Well it looks like it didn't do much, let's go back to the MAGT log and see what it did during this same time period:

[40076] (06/05 11:05:13.616):{0x920} {User} SendStatusToWirelessNetworkUsingSRP SendQ, TID=1911024
[40400] (06/05 11:05:13.616):{0x920} {User} Received datagram with content type OTAKEYGEN, TID=1911024, for user User
[40386] (06/05 11:05:13.616):{0x920} {User} {User} Sending message to device, Size=49, TID=7486040, TransactionId=-2070675455
[40081] (06/05 11:05:13.616):{0x920} {User} SendToWirelessNetworkUsingSRP - SendQ, TID=7486040
[30591] (06/05 11:05:13.616):{0x920} {User} {User} *** Activation *** transaction aborted.

OK, so it looks like when trying to generate an encryption key from the device, it goes through a mini Activation, however it aborts the activation, I am guessing, because it can't establish encryption with the device. So it's a catch-22.

Let's hook it up to the USB cable and provision the device & see what happens, theoretically this should just so the same things - resend the service books and regenerate the key - but do it over the wire. Let's look at the MAGT logs:

[30615] (06/05 11:09:33.130):{0x920} {User} Received RPC command [ProvisionUser] (id=1666)
[40000] (06/05 11:09:33.130):{0x920} {User} Synchronizing user configuration
[40000] (06/05 11:09:33.130):{0x920} {User} Synchronizing device capabilities
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Updated handheld capabilities info
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Handheld capabilities indicate AES encryption is supported
[40000] (06/05 11:09:33.161):{0x920} {User} ProvisionUser: Using key type AES
[40000] (06/05 11:09:33.161):{0x920} {User} Synchronizing AES encryption keys
[30000] (06/05 11:09:33.192):{0x920} {User} *** Activation *** User activated; triggering message pre-population
[30616] (06/05 11:09:33.192):{0x920} {User} RPC command [ProvisionUser] successful (id=1666)

Everything looks OK there, and now we can do lookups and send mail just fine.

What is different about the USB provisioning? Ideally I would like to be able to fix these types of problems OTA, in case someone is out of the office. I don't like the idea of having to hook up to the BES Manager and provision via USB to fix. Anybody out there know anything I could have tried OTA before hooking up to USB, without having to go through a full wireless enterprise activation?