On the heels of a Blackberry outage on December 22, 2010, several Blackberry Messenger bugs widely reported in June 2011, and the Blackberry outage in July 2011, comes a huge outage in October 2011.
The following quotes are taken from RIM’s UK service site.
Monday 10th October -15:00(BST)
We are currently working to resolve an issue impacting some of our BlackBerry customers in the Europe, Middle East and Africa region (EMEA.)
Tuesday 11th October -21:30(BST)
The messaging and browsing delays that some of you are still experiencing were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated and we are now working to clear that backlog and restore normal service as quickly as possible.
Wednesday 12th October – 12.00 (BST)
We know that many of you are still experiencing service problems. The resolution of this service issue is our Number One priority right now and we are working night and day to restore all BlackBerry services to normal levels.
Thursday 13th October – 17.05 (BST)
As BlackBerry services continue to return to normal, some users may still be experiencing delays with messaging and browsing. In some cases this can be resolved by a full device reboot. To reboot your device, remove and then reinsert the battery.
OK, so here are some of my thoughts on this outage:
The stated root cause was a core switch failure coupled with a failover mechanism that did not work. A third problem (it seems failures always come in quantities of three or more!) was that the system couldn’t handle the subsequent backlog of messages. In other words, it didn’t have the capacity to handle a large number of essentially simultaneous messages. Also, the system seemed to affect certain users in theUnited States. Perhaps this was because of backlogs of messages to or from EMEA. I couldn’t find an acknowledgement or explanation from RIM. I also find the level of disclosure from RIM to be far below the quality and completeness of Amazon’s disclosures that I reported on earlier. The bottom line is that the outage lasted from Monday afternoon to Thursday evening, or a little more than 72 hours, and it affected many users in EMEA and some inNorth America.
Aside from the fact that a 72 hour outage spoils one’s MTTR and availability numbers, it appears that the client base was a bit upset. This was not so much due to an outage, but it seemed to be due to a loss of confidence in RIM and its Blackberry system. Here are a few quotes posted on www.focus.com:
From Andrew Baker, Director, Service Operations,SWNCommunications Inc., posted onOct. 13, 2011,
“RIM has been having lots of PR problems and corporate customer problems of late.
“Based on my own personal sampling of several dozen customers who use RIM services, the sentiments generated by this set of outages are not good. There were a few hold-outs who were adamant that RIM would survive all the talk of doom and gloom, that are now looking to implement alternatives.
“This is at both the technical level and the executive level within these organizations.
Confidence has suffered considerably, and the timing for them could not be worse. And they totally botched the PR associated with this outage.
“They do not appear to have a sound strategy to deal with the many competitive challenges of their market, and they are poor communicates even in crisis. They failed to capitalize one of their core strengths, which was device security, and have undermined confidence in their other, which is their network.
“RIM is in the midst of a death spiral — the only question is how long it will take. Look for a number of highly publicized defections over the next few months, which will add fuel to the perception of their demise, and hasten it.”
From Patrick Adams, Director, Adduce:360: “In my experience the BB BES approach is still seen as the mainstream messaging solution for [corporations] – however, I do think that BB has a big challenge to maintain their position – irrespective of this recent outage.
From Adele Berenstein, Consultant and Trainer, Customer Satisfaction and Reputation Management: “I believe that RIM (the manufacturer of BB) has a unique position as the secure provider for email for corporations. Unless there is an alternative with equivalent functionality, corporations and governments will not have a choice but to forgive. I am sure the big corporations are putting pressure on RIM to fix their problems.
“When an alternative comes along, then RIM could conceivably loose their significant market share in the business and government markets.”
From Philip Sawyer, Managing Member, Voyage Media Group: “Honestly, I’m not sure RIM will ever get back on track. The well-established iPhone and Android devices have been steadily eating away at BB’s user base…and with Windows Phone rapidly gaining traction with their recent innovations, I think we’ll see another migration begin to take place, as die-hard business users begin to notice the growth of another trusted name in the corporate world (Microsoft) into the smartphone industry.”
And finally from Juan Barrera, “There is no way I am going back to BlackBerry, I waited over 5 years for them to come up with something remarkable, only saw excuses after excuses from its very-high-egos Co-CEOs. Now 100% Apple! at all levels.”
My personal opinion is that BB BES is a very nice product. I loved it when I was using it, in spite of a few quality problems. What is sad is to see RIM’s image erode with individual customers abandoning the BB, and with corporate customers looking for an alternative. With competition on the horizon, RIM should have been focusing on cementing their customer base. They clearly have not. The moral here is that becoming lax on quality and not working seriously to protect availability can really ruin a company.