Friday, March 6, 2009
Moving On...
Well today is my last day working on Domino systems, so I guess it is the last day for this blog as well. I have had fun probing the inner workings of Domino as it relates to BES, however all good things must come to an end, and I will soon be going back to the dark side. Adieu!
Thursday, January 8, 2009
Software Configuration Push Problems Wrapup
Well here is the upshot of pushing an application to ~1500 Blackberrys: About 50 of them will not receive the app for a variety of reasons. (Which is actually not too bad considering the complexity and fragility of all the pieces involved in a wireless software deployment!)
First, check the Last Contact Time for the user's BES account. I have about 8 of these guys who haven't been heard from in weeks or months, they are probably on maternity / paternity leave or have been fired but not deleted from the BES. Ignore these guys for now.
Second, ensure there is enough memory on the device via the BES Manager. If the device reports 1MB free then it probably won't install the app. If the last date status was reported is way in the past, the device probably doesn't have enough memory to even report daily status anymore.
Then, check that the device's Hardware information is reported in the BES Manager. Some devices only have there "Secured Boot ROM" entry there, which means the hardware is not reporting correctly, and is reporting a hardware ID of 0x00000000. The BES doesn't see this number in the device.xml and won't push the app to it. This happens alot with Verizon devices, especially the 8300 models, but I have seen a few with T-Mobile 8700's as well. In this case the only thing I have discovered that fixes it is to upgrade to a later OS, such as 4.5.
If memory is plentiful and hardware info is there (and of course your device.xml is updated), then I have found that there may be a mismatch between the Applications / Modules reported by BES Manager and what is actually on the device. Many times the BES Manager will report multiple modules with different version numbers, but the device is actually just fine. In this case we need to clear the BES Manager device info. The best way to do this is to go into the PIM Sync Options of the user and set the Automatic BlackBerry Device Management Enabled setting to False, clicking OK, waiting 1 minute, then going back in and setting it to True. This will clear the data from the SQL tables, and when set back to True, will request that the device repopulate the information with fresh data. When this is done, after a few minutes, often the software push will begin to work.
On some devices I set the above setting back to True and I never get back any device information! In this case the Desktop (SYNC) service book on the device seems to be corrupted, and having the user delete it and undelete it restores everything just fine. Then the push works.
Please note that deleting the user's BES account, recreating it, wiping the device, and reactivating will most likely fix all the above issues. However in our environment it is imperative that our attorneys have as little down time as possible, so fixing these issues behind the scenes is a lifesaver.
Hope this little learning experience helps you all out there as much as it has me.
First, check the Last Contact Time for the user's BES account. I have about 8 of these guys who haven't been heard from in weeks or months, they are probably on maternity / paternity leave or have been fired but not deleted from the BES. Ignore these guys for now.
Second, ensure there is enough memory on the device via the BES Manager. If the device reports 1MB free then it probably won't install the app. If the last date status was reported is way in the past, the device probably doesn't have enough memory to even report daily status anymore.
Then, check that the device's Hardware information is reported in the BES Manager. Some devices only have there "Secured Boot ROM" entry there, which means the hardware is not reporting correctly, and is reporting a hardware ID of 0x00000000. The BES doesn't see this number in the device.xml and won't push the app to it. This happens alot with Verizon devices, especially the 8300 models, but I have seen a few with T-Mobile 8700's as well. In this case the only thing I have discovered that fixes it is to upgrade to a later OS, such as 4.5.
If memory is plentiful and hardware info is there (and of course your device.xml is updated), then I have found that there may be a mismatch between the Applications / Modules reported by BES Manager and what is actually on the device. Many times the BES Manager will report multiple modules with different version numbers, but the device is actually just fine. In this case we need to clear the BES Manager device info. The best way to do this is to go into the PIM Sync Options of the user and set the Automatic BlackBerry Device Management Enabled setting to False, clicking OK, waiting 1 minute, then going back in and setting it to True. This will clear the data from the SQL tables, and when set back to True, will request that the device repopulate the information with fresh data. When this is done, after a few minutes, often the software push will begin to work.
On some devices I set the above setting back to True and I never get back any device information! In this case the Desktop (SYNC) service book on the device seems to be corrupted, and having the user delete it and undelete it restores everything just fine. Then the push works.
Please note that deleting the user's BES account, recreating it, wiping the device, and reactivating will most likely fix all the above issues. However in our environment it is imperative that our attorneys have as little down time as possible, so fixing these issues behind the scenes is a lifesaver.
Hope this little learning experience helps you all out there as much as it has me.
Tuesday, December 23, 2008
Software Configuration Push Problems cont'd...
Another issue I discovered is the following:
1. I audit all of our devices to ensure the memory is above 3MB before pushing the software, to avoid causing their devices to delete all their mail. How to do this is another post in itself!
2. We send these users emails on how to clean up the device.
3. I got some emails back saying they had plenty of memory free already!
4. I checked and found that the Device information had not been updated in 6 months or more! I was working with old data for some users!
Here is my theory:
Once you get to a very small amount of memory on the BB, the Handheld Agent will stop reporting back to the server it's details on a daily basis. Fortunately there was an easy fix for most of these guys, and that is to set "Automatic BlackBerry Device Management Enabled" to FALSE, wait a few minutes, then set it back to true. What this does is delete all the device information for that user from the SQL database, then when you turn it back on it refreshes it directly from the device. If there is enough memory available you will get new info. If memory is still low you will not get any info back.
Interesting side note: The refreshed device information will only reflect when the Handheld Agent last collected the data, not the "right now" state. So if you get todays info it may be from 5 or 7AM, not from 11AM when you changed the settings.
1. I audit all of our devices to ensure the memory is above 3MB before pushing the software, to avoid causing their devices to delete all their mail. How to do this is another post in itself!
2. We send these users emails on how to clean up the device.
3. I got some emails back saying they had plenty of memory free already!
4. I checked and found that the Device information had not been updated in 6 months or more! I was working with old data for some users!
Here is my theory:
Once you get to a very small amount of memory on the BB, the Handheld Agent will stop reporting back to the server it's details on a daily basis. Fortunately there was an easy fix for most of these guys, and that is to set "Automatic BlackBerry Device Management Enabled" to FALSE, wait a few minutes, then set it back to true. What this does is delete all the device information for that user from the SQL database, then when you turn it back on it refreshes it directly from the device. If there is enough memory available you will get new info. If memory is still low you will not get any info back.
Interesting side note: The refreshed device information will only reflect when the Handheld Agent last collected the data, not the "right now" state. So if you get todays info it may be from 5 or 7AM, not from 11AM when you changed the settings.
Software Configuration Push Problems (SCPP's)
Ok here I am again after a long bout of laziness. Anyway I am working on deploying a piece of software to all our devices and am running into fun issues. Here is one of them:
8330 Curves on Verizon running 4.2 or 4.3 OS, for some reason, do not publish their hardware details to the BES. In fact they don't even publish it if you are connected over USB to the device via BlackBerry Manager. The Hardware ID is just 0x00000000.
This is a problem because when you push software it checks the device model against the device.xml file in the AppLoader directory above the actual software package. If it doesn't find it then you get an error in the POLC log about HHCM_DEVICE_NOT_SUPPORTED and to Update your device.xml.
The issue of course is not the device.xml, but that the Hardware ID is not there for this device.
I found that upgrading from 4.2 or 4.3 OS to 4.5 will publish all the gory hardware details and software pushes will work again.
I wonder if it is stupid Verizon wanting to hide the hardware from everyone but themselves (like they do with GPS) or just an bug in their version of OS 4.2 / 4.3. Either way seems it is resolved in 4.5.
8330 Curves on Verizon running 4.2 or 4.3 OS, for some reason, do not publish their hardware details to the BES. In fact they don't even publish it if you are connected over USB to the device via BlackBerry Manager. The Hardware ID is just 0x00000000.
This is a problem because when you push software it checks the device model against the device.xml file in the AppLoader directory above the actual software package. If it doesn't find it then you get an error in the POLC log about HHCM_DEVICE_NOT_SUPPORTED and to Update your device.xml.
The issue of course is not the device.xml, but that the Hardware ID is not there for this device.
I found that upgrading from 4.2 or 4.3 OS to 4.5 will publish all the gory hardware details and software pushes will work again.
I wonder if it is stupid Verizon wanting to hide the hardware from everyone but themselves (like they do with GPS) or just an bug in their version of OS 4.2 / 4.3. Either way seems it is resolved in 4.5.
Wednesday, May 7, 2008
ITAdminQueue - EXPOSED!
A pretty cool new feature appeared in 4.1 SP5, resulting in a new tab called Software Configuration Status within the BlackBerry Manager application:
This tab shows a view of the previously hidden ITAdminQueue table in the SQL database, which contains all sorts of back end commands that are queued up / in process to the BlackBerry handhelds.
The view above results from setting the software configuration of my test BlackBerry to a pre-configured RSA SecurID application, turning the wireless off, and manually pushing it out via the "Deploy Software" command. You can see there are two entries listed under this new tab, which don't show a whole lot of information. I assume because they are simply queued, and have not errored out yet.
Here are the corresponding ITAdminQueue rows in the actual SQL table:
Note that the first entry is actually listed with a Command column number of 129. This corresponds to the BES Policy service's SEND_APC_ITPOLICY command. The IT Policy is always sent first whenever an application is delivered, as it includes any software application policies you have created. As these policies need to be applied before the app is installed, sometimes there are errors applying the IT policy - which can prevent an app from being delivered successfully.
Note also that this IT Policy command does not show up in the Software Configuration Status tab, which leads me to believe this tab is built using a filter to only show software application entries, not the whole table.
Below this row there are two additional rows listed, both showing the Command column set to 128. 128 corresponds to the SEND_APC_APP command, so these are the commands to send the application itself. Note that these rows in the SQL table do not have a Time Sent value, but the IT Policy does. More proof that the app will not be sent until the IT Policy is applied successfully.
One interesting thing is that the app I was pushing contained only one .cod file, so why two SEND_APC_APP commands / rows? Looking into the Policy log we find something interesting:
Queuing the package RSA SecurID, consisting of 1 modules in 2 parts.
So I think this this one .cod module is split into 2 parts for transmission to the BlackBerry, for what reason I do not know. Perhaps it was too big to send all at once?
Although this new tab of information is helpful, I see it does not allow you to delete any queued commands. Some software push issues have been resolved by running the following SQL script:
osql -S [SQLServerName] -d [DBName] -U [SQLLogin] -P [SQLPassword] -Q "delete from ITAdminQueue where (command = 128 or command = 129) and status = 6"
This command will clear all SEND_APC_ITPOLICY and SEND_APC_APP entries with a status of 6, which designates an error; this allows future pushes to proceed properly.
Some other BES Admins have recommended just deleting all rows referencing a particular PIN number to resolve issues with an individual BlackBerry not receiving an application properly. I cannot vouch for this method however, as I have not used it. I do occasionally run the SQL script above during standard maintenance of our BES environment, or if a particular application push problem occurs.
From RIM's documentation, here are some error codes you might see in this new tab:
* Device timed out waiting for module
* Device reported insufficient memory to install module
* Device reported insufficient privileges to install module
* Device reported Invalid Version in packet, supported version is %s
* Device reported that there was an Incomplete Module
* Device reported that the Module Save Failed
* Device reported a general failure installing the module
* Incomplete ACK data for APPD request
* Device reported a %s error while installing module
* Device Reported Data Format Error in packet while installing module
* Device Reported Invalid Command while installing module
* Device Reported Insufficient Body Data while installing module
* Device Reported Invalid Module Hash while installing module
* Device Reported Invalid App Data Length while installing module
* Device Reported Insufficient App Data while installing module
You can check out the support article, KB15470, for more information on these messages and there resolutions.
I am sure that this new tab will help in resolving future application push issues, however I am also sure that some manual cleaning of the ITAdminQueue table will still be needed occasionally as well!
This tab shows a view of the previously hidden ITAdminQueue table in the SQL database, which contains all sorts of back end commands that are queued up / in process to the BlackBerry handhelds.
The view above results from setting the software configuration of my test BlackBerry to a pre-configured RSA SecurID application, turning the wireless off, and manually pushing it out via the "Deploy Software" command. You can see there are two entries listed under this new tab, which don't show a whole lot of information. I assume because they are simply queued, and have not errored out yet.
Here are the corresponding ITAdminQueue rows in the actual SQL table:
Note that the first entry is actually listed with a Command column number of 129. This corresponds to the BES Policy service's SEND_APC_ITPOLICY command. The IT Policy is always sent first whenever an application is delivered, as it includes any software application policies you have created. As these policies need to be applied before the app is installed, sometimes there are errors applying the IT policy - which can prevent an app from being delivered successfully.
Note also that this IT Policy command does not show up in the Software Configuration Status tab, which leads me to believe this tab is built using a filter to only show software application entries, not the whole table.
Below this row there are two additional rows listed, both showing the Command column set to 128. 128 corresponds to the SEND_APC_APP command, so these are the commands to send the application itself. Note that these rows in the SQL table do not have a Time Sent value, but the IT Policy does. More proof that the app will not be sent until the IT Policy is applied successfully.
One interesting thing is that the app I was pushing contained only one .cod file, so why two SEND_APC_APP commands / rows? Looking into the Policy log we find something interesting:
Queuing the package RSA SecurID, consisting of 1 modules in 2 parts.
So I think this this one .cod module is split into 2 parts for transmission to the BlackBerry, for what reason I do not know. Perhaps it was too big to send all at once?
Although this new tab of information is helpful, I see it does not allow you to delete any queued commands. Some software push issues have been resolved by running the following SQL script:
osql -S [SQLServerName] -d [DBName] -U [SQLLogin] -P [SQLPassword] -Q "delete from ITAdminQueue where (command = 128 or command = 129) and status = 6"
This command will clear all SEND_APC_ITPOLICY and SEND_APC_APP entries with a status of 6, which designates an error; this allows future pushes to proceed properly.
Some other BES Admins have recommended just deleting all rows referencing a particular PIN number to resolve issues with an individual BlackBerry not receiving an application properly. I cannot vouch for this method however, as I have not used it. I do occasionally run the SQL script above during standard maintenance of our BES environment, or if a particular application push problem occurs.
From RIM's documentation, here are some error codes you might see in this new tab:
* Device timed out waiting for module
* Device reported insufficient memory to install module
* Device reported insufficient privileges to install module
* Device reported Invalid Version in packet, supported version is %s
* Device reported that there was an Incomplete Module
* Device reported that the Module Save Failed
* Device reported a general failure installing the module
* Incomplete ACK data for APPD request
* Device reported a %s error while installing module
* Device Reported Data Format Error in packet while installing module
* Device Reported Invalid Command while installing module
* Device Reported Insufficient Body Data while installing module
* Device Reported Invalid Module Hash while installing module
* Device Reported Invalid App Data Length while installing module
* Device Reported Insufficient App Data while installing module
You can check out the support article, KB15470, for more information on these messages and there resolutions.
I am sure that this new tab will help in resolving future application push issues, however I am also sure that some manual cleaning of the ITAdminQueue table will still be needed occasionally as well!
Tuesday, May 6, 2008
Heading to WES 2008
Well I know some of you who read this blog will be at WES this year, looking forward to meeting all of you in addition to the great bunch of BES Admin folks that convened last year.
Generally our meeting place was in the right rear of the solutions showcase but we may also be hanging out around the BoxTone lounge, check out the WES 2008 thread on www.blackberryforums.com BES Admin corner for the latest details...
Hoping to see you there!
Generally our meeting place was in the right rear of the solutions showcase but we may also be hanging out around the BoxTone lounge, check out the WES 2008 thread on www.blackberryforums.com BES Admin corner for the latest details...
Hoping to see you there!
Monday, May 5, 2008
Address Book Contact Labels and the BlackBerry
Had an interesting issue crop up today with an end user who changed the contact labels associated with an address book entry. Here is the default when you click the Edit Contact Labels button within your address book entry form:
This person had modified the labels to reflect different information, for example changing the Pager label to Home Phone #2. The problem is that the label change does not sync over to the BlackBerry, which shows the Label still listed under Pager.
I called RIM and verified this is the way it should work, as the underlying field name does not change, and RIM maps to the field name, and not the changeable display name.
I understand the issue and agree with RIM's assessment, however this is one of those gray areas where it is difficult to determine who is responsible for the issue. Is it Lotus for providing the (perhaps unwise) ability to change display names to something that is completely different from what the underlying field name reflects? Or is it RIM for not supporting this Lotus Notes feature, even though it may not be the best of features to have from a data integrity standpoint.
As is generally the case, it is perhaps a little of both, which is why these things tend not to get fixed as each side points to the other and neither will take the responsibility to address it.
This is not that big of a deal to me, and personally I agree with RIM's assessment that their sync engine should map to the field name and not a nebulous display name that can be changed at will by the end user.
However, this is just a small example of the types of conflicts that can happen when you have two separate software entities interacting with each other, creating a no-man's land of responsibility with the customer trapped in the middle.
This person had modified the labels to reflect different information, for example changing the Pager label to Home Phone #2. The problem is that the label change does not sync over to the BlackBerry, which shows the Label still listed under Pager.
I called RIM and verified this is the way it should work, as the underlying field name does not change, and RIM maps to the field name, and not the changeable display name.
I understand the issue and agree with RIM's assessment, however this is one of those gray areas where it is difficult to determine who is responsible for the issue. Is it Lotus for providing the (perhaps unwise) ability to change display names to something that is completely different from what the underlying field name reflects? Or is it RIM for not supporting this Lotus Notes feature, even though it may not be the best of features to have from a data integrity standpoint.
As is generally the case, it is perhaps a little of both, which is why these things tend not to get fixed as each side points to the other and neither will take the responsibility to address it.
This is not that big of a deal to me, and personally I agree with RIM's assessment that their sync engine should map to the field name and not a nebulous display name that can be changed at will by the end user.
However, this is just a small example of the types of conflicts that can happen when you have two separate software entities interacting with each other, creating a no-man's land of responsibility with the customer trapped in the middle.
Subscribe to:
Posts (Atom)