Domino server is crashing – SMSDOM panic

This happened just today.  Installed a brand new Domino 8.5.1 server 64 bit.  Installed the latest Symantec anti virus for Domino on top of it.  Ran a manual scan and crashed the whole server when completing the scan of a particular mail file.

The very reproducible crash would generate an SMSDOM Panic error with a rather generic error message of “Entry not found in index”.

When Google yielded no appropriate results, we called Symantec and they immediately pointed us to this technote.

Apparently, this is a well known problem with a simple fix.  Follow the steps in the technote to replace the nntask.exe and life, once again, is good.

Lotus must get customers to upgrade

As we’re going into 2010, Lotus is faced with numerous customers evaluating their email infrastructures.  While Lotus reports record-setting rates of upgrades to the latest version of Notes and Domino (8.5), large numbers of customers are still running older versions of the Lotus software, with some of them going as far back as release 5 desktops and mail templates.  Lotus knows that customers on versions earlier than 8.5 are more likely to migrate to a different mail solution (Exchange and Outlook) as compared to customers who went through the 8.5 upgrade. The R5 mail template wasn’t all that pretty when it came out 10 years ago.  It looks absolutely horrible when you compare it Outlook 2007 and the likes.  Lotus’ latest release of the Notes client 8.5 looks and feels so much like Outlook and offers so much functionality that it makes it hard to defend the migration argument and the associated costs.

What Lotus needs to think of now is how to encourage its customers to upgrade.  Over the course of 2009 a great deal of effort and money — sales activities, SWAT teams — was expended on defending Lotus at existing customers.  Often these very costly efforts were not successful and the battle was lost.  Perhaps the efforts were pointed in the wrong direction.  Offense is the best defense.  Instead of trying to convince customers to stays, Lotus should make staying so appealing that customers don’t even entertain the thought of leaving.  Instead of defending against Microsoft, Lotus should look at making the 8.5 upgrade financially appealing to the customers by offering deep discounts on license renewals under the condition that within 6 to 9 months every Domino server and every Notes client will be upgraded to 8.5.x.  The somewhat successful V2V (Version To Version) campaign conducted at the end of 2009 could’ve been more successful with a simple phone call to the CFO: “How much will you be spending on your license renewals with IBM this year? How would you like to spend 50% less?”

This might be a hard pill to swallow for the sales force.  The initial impression is that they would be losing 50% of their commission and making 50% of their quota.  To address the concern and to keep the sales teams motivated, IBM could apply the customer discount after the commission and the quota fulfillment have been calculated.  And to overcome all internal objections, IBM can treat this offer as a competitive upgrade price situation.

This practice is not unique.  It is widely practiced by consumer services companies in markets where there’s a lot of competition.  Cell phone carriers, TV cable or satellite providers and even some credit card companies routinely offer credits and discounts to long-time customers who are considering canceling their contract and taking their business to a competitor.

Before it’s too late, IBM, should figure out how to get the existing customers to upgrade to 8.5 and upgrade quickly.  Making it a financially appealing decision, is a good first step.

MWLUG is coming next week

There’s a bit more than 1 week left before MWLUG kicks off in Chicago on August 27th.  If you haven’t registered yet, do so and do so soon.  If you live in Midwest and particularly around Chicago and are a Lotus user and fan, you almost have no reason not to attend.  For myself, I’m looking forward to attending a conference where I don’t have to travel out of town.

PSC, while sponsoring the event, will be presenting 3 separate sessions.

Integration and Coexistence: Leveraging Lotus and Microsoft products to build better solutions” by John Head.  While John and I usually co-present this session at Lotusphere, the 1-hour format of LUG conferences makes it kind of silly to have 2 presenters up there.  So this will be John all himself.

Enabling your Applications with Complete Rich Text Editing on the Web” by yours truly.

Implementing DAOS and ID Vault” by Luis Guirigay.  Luis will spend the 2 days prior to MWLUG teaching a Domino 8.5 upgrade workshop.  So all topics of new and exciting features of 8.5 will be fresh on his mind.  Come by and ask questions.

Hope to see you all there.

Domino 8.5 in Omaha – Part deux

All is well that ends well. Well, it hasn’t quite ended just yet. But, damn!, I’m good. Alex, even though you’ve been a manager for the past 3 years, you still got it! After several days of going back and forth with IBM tech support, where they just never seemed to be able to gather enough information to diagnose the problem, I found an SPR that described the very same problem, with exact same symptoms and the very same resulting error message. The SPR was supposed to have been fixed i the Domino 8.0.1 code stream. But here it was again, an apparent regression bug of sorts, causing the poor server to hang every morning. Basically, the router was getting into a lock situation trying to access one of the files and running an LZ1 compression algorithm.

Unfortunately, this thing doesn’t appear to be easy to solve. We were hoping that the development team would be able to put together a hot fix in a matter of hours, after all, they did solve it once before. However, they confirmed that the fix for that SPR is in the 8.5 code stream. So now we’re waiting, with baited breath, to hear what the development team comes back with. Come on, Westford folks, make me proud!

And while we’re waiting, a few interesting observations while working with the IT team of this particular client, if I may.

For reasons unknown to me, this organization is on its 3rd mail system. Once upon the time, they were an Exchange shop. Then later they moved to GroupWise. And now they are a Lotus shop. While I am not completely clear on their decision to move to Domino, some part of that decision was based on their ability to run mail server on top of AIX. Some years ago, AIX was my favorite platform to run Domino on: it was rock solid, required minimal maintenance and ran well. These folks did their homework. They had talked to a number of companies that ran Domino on AIX and everyone confirmed that AIX and Domino is a winning combination.

They told me of one company they spoke to, which used to run Domino on the iSeries (i5 OS, I believe). It ran so poorly that the company decided to move off Domino to Exchange. Then, somewhere along the process, they tried moving Domino on to AIX and it ran so well, that they canceled their Exchange migration and stayed with Lotus, although now on top of AIX.

Having said that, some of the younger members of the IT team did tell me that they didn’t like the Notes client. They thought it wasn’t as user friendly as even the latest version of GroupWise that they ran. They readily acknowledged that Domino ran better and did things better than GroupWise; that a lot of email problems that they had with GroupWise simply went away when they moved to Domino. But they still complained of usability of the Notes client.

This is nothing new.  I’ve been saying this for quite some time now. IBM makes great servers. Lotus makes a great server, there’s probably none better. Their client though is not so much. How I wish that IBM would support other email clients with full fidelity. This client in Omaha is a great case study of that. The admins love AIX, the admins love Domino, let the users run whatever they want to run: Notes, Outlook, Mac Mail, Thunderbird, whatever else. And may system administrators and end users exist in perfect peace and harmony. Like the famous bumper sticker says: “COEXIST”.


Domino 8.5 in Omaha

It is not often nowadays that I get to spend several days on a full time billable consulting engagement.  And when the said engagement takes me on the road to Omaha, Nebraska and constitutes helping a brand new Lotus Domino client, I just have to write about it.

A company here, in Omaha, very recently (2 weeks ago, to be exact) migrated from Groupwise to Lotus Domino 8.5  (Hip, Hip, Hooray and 3 cheers for their decision to go with Lotus.)  They took the plunge and deployed all the latest and greatest features: DAOS, Transaction Logging, ID Vault, SameTime, Quickr — the works.  To top it off, they deployed it on AIX.  Some kick-ass boxes: 12 CPUs, 12 GB of RAM — the thing screams.

Unfortunately, I have a bad feeling that the number of Domino 8.5 customers running on AIX can be counted on the fingers of one hand of a Chinese butcher.  In these 2 weeks they’ve had nothing but problems with server up time, stability and performance.  So several hot fixes later, things are beginning to look better.  I’m here to help troubleshoot, to work with IBM support and to validate their setup and configuration as left by another business partner.  

After a very stressful day of the primary mail server going up and down like a yo-yo every 2 to 3 minutes, I just can’t wait to see what tomorrow morning will bring.  The server was finally stable at the end of the day.  This same server freezes up every morning between 7:15 and 7:30 am.  What are the chances that whatever I did today will solve that problem, too?

Notes 8.5 Upgrade Workshop in June

The upgrade workshop that we hosted at the of May was a huge success.  We ended up having to turn people away.  As the result, due to popular demand, so to speak, we are hosting another workshop at the end of June.

On June 24th and 25th we will host a Notes/Domino 8.5 upgrade workshop in Schaumburg at PSC.  Detailed information on the workshop and registration is on the PSC website.  So, if you didn’t get a chance to make it to the first workshop, hope we’ll see you on the 24th.

MWLUG conference 2009 registration opens today

The registration for 2009 Midwest Lotus User Group Conference opens today.  For more information on the conference and to register visit the MWLUG site.  

And don’t forget to register for the Notes & Domino 8.5 upgrade workshop to be held in the 2 days prior to the start of the conference.

Notes 8.5 upgrade workshop

On Wednesday and Thursday of this week we [PSC] and IBM hosted a Notes 8.5 upgrade workshop. The workshop was held at IBM Innovation Center downtown Chicago and it was a great event. Thank you everyone who found 2 days in their busy schedules to join us for the workshop. The event was “sold out”. We had no empty seats in the room and unfortunately we had to turn away a few people who tried to register at the last moment. Not to worry, we are planning to hold more workshops during the summer. Please come to the PSC site for event information and registration.

The workshop focused on showcasing the new and exciting features of Notes and Domino 8.5: things like DAOS, ID Vault, Shared Logon, XPages. The agenda is a good mix of presentations and hands-on labs, so not only did the attendees get to listen about the new features, they got to feel and touch them, practice setting them up and using them.

All in all, a great event. I’m looking forward to hosting a few more of these during the summer.

Currently, we’re looking at hosting 1 workshop in Schaumburg at the PSC office on June 24 – 25.

In addition, we’re planning to host another workshop downtown Chicago at the IBM center just before MWLUG on August 25 – 26.  Seating will be limited again, so please stay tuned for registration announcements.