Sales-sold dream results in big bill for fix • The Register
Before shelling out big bucks to hire the services of consultants, perhaps consider this week’s episode of Who, Me? and the adventures of a reader who will be called “Norman” for the time of this tale.
Towards the end of the 2000s, Norman left an in-house software development “to join the ranks of The Consultants”.
“My technical credentials were solid…assuming The Consultancy gave me projects that really needed them,” Norman explained.
And indeed, during his first job as a Consultant, he was asked to work on “a technology in which I had at best a passing experience: – Microsoft Exchange.
The consulting firm did not care that he was not a stock market expert. In Norman’s account of the story “The sellers had done an excellent job of selling The Dream and it had a marginal consequence for them that I had next to no idea what I was doing and certainly not the experience to foresee some of the things that could go wrong”
“I had a sharp new suit and a misplaced sense of confidence and superiority, so I was ready.”
Norman’s consulting assignment was to write a service that monitored an Exchange mailbox and forwarded certain messages to a document management system.
“The business logic was relatively simple – if the email provided the required information in a specified format, then it was sent to the DMS; if the required information was missing, the email was returned to the sender with a helpful error message.
Norman wrote the required code, which “went through user acceptance testing with flying colors and was cheerfully deployed into production”.
Until it breaks.
Norman had long since been redeployed, but luckily he was still with the same client.
He was therefore able to observe “an absolute pandemonium as the entire IT team complained savagely that their entire Exchange infrastructure was on its knees and no one could receive emails”.
That wasn’t Norman’s problem. But as the IT team was sorting through the mess, he heard a phrase that suggested the mailbox processing service he had built might be the culprit.
A little investigation proved that his work was indeed the cause of the problem – but not because Norman had messed up.
That honor went to a business unit of Norman’s client, who had created a new automated process that sent emails to the mailbox monitored by the service he had created. But the automated process misconfigured the emails so the required information was missing and added an autoresponder on the mailbox they used to send the emails.
So a game of answering machine tennis ensued.
“Their mailbox had sent ‘invalid’ mail to my mailbox,” Norman explained. “My service had found the message and resent it. And their autoresponder responded with an equally invalid message. My service responded.
And how he responded! Instead of just deleting incoming messages from the mailbox, it rejected them. This meant that whenever Norman’s service was running, it wasn’t just processing the last mail… and all previous posts.
Norman described the result as “an exponential letterbox time bomb that was enough to completely overwhelm the entire environment”.
Norman fixed the problem by suspending the mailbox monitoring service, purging the target mailbox, and temporarily disabling the autoresponder on the other mailbox.
That done, he was triumphant. “I had successfully solved the problem…that my code had created,” he admitted.
And then he had another win. “The consultancy has staged another big sale in the form of the ‘change request’ needed to implement a two-line code fix to prevent this from happening again.”
Isn’t the council great?
If you’ve ever charged a client to make them pay for your own mistake, or paid to fix a consultant’s incompetence, say so by emailing Who, Me? ®