About two weeks ago, on Wednesday, August 5, I had an “unfun” conversation. One that I and every engineer here at WebWorks.com labored to avoid. Our contract licensing system encountered a glitch and a customer was left in a lurch.
Where are my keys!?!
Back in 2008, the Development team was tasked by management to improve the customer experience for ePublisher license management. The primary goal for the licensing system was to ease the administrative burden for customers and ourselves associated with our quarterly release system. Oh, and it should never, ever leave a customer in a lurch. But, as they say, “Hang-ups and Bang-ups can happen to you“.
We did run into a localization issue just after 2009.1 shipped, one that required a bit of creative thinking to resolve. Yet by the end of May, we were all fairly amazed that the contract licensing transition had gone so smoothly. We ran weekly reports, looking for any errors reported by the licensing system, and contracted customers who might encounter issues. There were small items for us to tweak here and there. Nothing major.
So imagine our surprise when, three months after deployment, we encountered a completely unexpected situation. Suddenly, an event, an event which had executed successfully thousands upon thousands of times in those three months, failed. Nothing literally blew up. Even so, the licensing system had just done something that, by design, should have been impossible.
The good news? The customer was able to manually update ePublisher's license keys and make a scheduled delivery.
The bad news? Now that we could see the pattern, this event had been occurring consistently over the past three months. It was just that the system never failed twice in a row (built-in retry mechanisms kicked in!). However, if the system encountered two succeeding errors, ePublisher happily unregistered the user and deleted the installed keys!?!
After analyzing the problem, we determined changes are required in ePublisher's client-side code. Great! But such a change won't help users of our 2009.1 and 2009.2 releases. So, we did a bit more thinking, found a way to address the problem on our side of the fence, and join the high fliers who soar to high heights.
So what have I learned from this experience? Perhaps I didn't learn a lesson so much as find a reminder.
One can never choose when and how to meet adversity. All one can do is choose the response.
We'll just commit to continuing to work hard and respond to whatever challenges come our way.