"It becomes a game of tactical cat and mouse," says Jacobson. "You're not attacking anything with real focus."
Stefan Estrada, engineering manager, Verizon "Here's the litmus test: If you're spending more time on bugs than you are on functionality, then you need to rethink the implementation and start over."
--Stefan Estrada, engineering manager, Verizon
Decide what's important, say our pros, and don't get lost in gradients of importance.
Jacobson explains: "I was having a conversation with one of my peers who was saying, 'We need to add this to the priority list because it keeps getting pushed down and not executed.' And I said, 'It's clearly not important; let's not add it to the priority list. If it were important, we'd be doing it. Or if it is important, then we have to stop doing everything else.'"
In some cases, managers see an outside firm as the solution to fix the problem, but consultants can compound the errors if the software's purpose isn't clearly articulated by those doing the hiring. The company keeps throwing money at the problem -- when the solution isn't about cost.
"The consultants will do what they're hired for," says Estrada, "but because there's not a clear vision as to what they're supposed to be doing -- say, they're adding more features, but that might not be the end goal of what the company is trying to achieve. You need a good set of program managers who can communicate between all the groups to make sure the right functionality is being created."
The total teardown
Sometimes a rehab job isn't a rehab job at all. The foundation has obvious cracks, and everybody can see them. The architecture is so poorly designed it needs to be scrapped and started anew. But how do you know when it's time to toss the entire project? Why not stick with code you have and exterminate the bugs?
"Here's the litmus test: If you're spending more time on bugs than you are on functionality, then you need to rethink the implementation and start over," says Verizon's Estrada. "If it's a complete patchwork that needs constant maintenance, then you're wasting a lot of time."
It's a daunting task to throw out the old and start anew. Not everybody is going to be happy about it. If a demolition is necessary, our pros say communicating that effectively to the various workgroups is the first step.
"Within the first couple of months of being at Netflix," Jacobson recalls, "I basically said, 'This application that all of you are using, we're going to throw that away. We're going to go with a new approach.'"
But ditching the system immediately isn't usually an option. In Jacobson's project, too many legacy devices counted on the existing platform and it would take time to transition to the new tools.
Sign up for CIO Asia eNewsletters.