A dark cloud. And its silver lining.
vaidyNovember 23, 2015
One of our more challenging consulting assignments was to build a Mobile Device Management (MDM) solution from the ground up. It’s a network of 5000+ devices- android tablets and linux based video players, installed across the US in hundreds of cities.
These devices would be loaded with health content and installed at health clinics - mostly educational videos about specific ailments. And patients consume this content when they visit these clinics.
We built the software that allowed anyone to control these devices remotely. It was complex. You could control everything on these devices - what videos they played, the volume, the brightness, interspersing ad content between videos etc
[Click the image below to watch the video.]
A suspicious “Virus”
Things were going well, when one day, we got a Red Alert email from our customer. Some of the devices were not accessible from the command centre. This kind of problem usually corrected itself, when the internet connection at the clinic got back to normal.
But this time it was different. It was almost like a virus - spreading across the entire network, one device at a time.
The Cause
To cut a long story short, the devices were left with zero disk space to function. Some software we had installed on these devices was eating up the space.
Unfortunately none of the techniques we tried to remotely resurrect these devices worked. The only option left was to send technicians to the clinics and run the software fix on them locally. And this was costly. Very costly. :(
Safety Valves
There were supposed to be two safety valves:
The first was a test device we used to simulate our code in. Why did we not see the problem on that one? Turns out it was much newer than the dead ones on the network. And new devices have more disk space. A lot more.
The second was a monitoring system we were building to check on things like disk space. The only problem? It was still work-in-progress, almost ready to prevent this exact kind of nightmare.
When the chips are down…
This is not a great story. It didn’t end well. We worked round the clock that entire week supporting the technicians while they fixed device after device.
That was not a fun week. None of the usual banter and humor that you’d otherwise see. But there was also no finger pointing. No arguments. No grumbling. Just silent work.
And this is what we’d like to call teamwork. Resilience in the face of intense adversity. The maturity to empathize with your team members and take collective responsibility. To help everyone focus on the job at hand. To introspect silently.
The bright side
Talking about this story is not easy. But it’s a good reminder that tough times almost always bring with them something good as well.
For us, it was the increased trust across the team. When your colleagues become your friends and your supervisors become your mentors - that’s a nicer workplace.
How about you and your team? Is this something that resonates with you? Would love to hear your stories too. :)