Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Real-world devops failures -- and how to avoid them

Bob Violino | June 28, 2016
To deliver on the promise of devops, heed these hard-earned lessons of devops gone wrong

The agency was launching an important project to build a web application. "As the vendor responsible for the ALM [application lifecycle management] process, we set out to establish tooling and processes covering definition and planning, code and commit, and build and release, all done in a collaborative, open source-inspired manner," Dawson says. 

The deployment and configuration of the supporting devops tooling was successful, Dawson says. "Unfortunately, devops cannot be implemented strictly with tools alone," he warns. "Devops requires equal attention to people or culture, process, and tools."

The project involved multiple teams on a tight, fixed deadline, leading management to seek the quick fix and focus primarily on the tools platform. "We were able to build a platform which included robust agile planning tools, a modern SCM [software configuration management], and Jenkins for continuous integration all deployed on a somewhat elastic, scalable platform."

However, the agency largely ignored the people and process portion of devops, and failed to gain the buy-in from developers and other stakeholders that was needed to build a devops strategy that would actually be put to use.

"This meant that though we had a ‘devops platform' in place, it was effectively used to support the same old legacy practices," Dawson says. "Developers deferred commits, merges, and integration; automated QA [quality assurance] and release were never fully implemented; broken builds were no big deal, and production loads in production-like environments were never tested."

When the client released the web application it immediately experienced critical and very public failures, as it hadn't been regularly tested in a production environment or by real users. In addition, once the problems became apparent, it took the agency multiple, multi-week development cycles to fix the issues and get the site operational. The slow response times served to exaggerate the impact of the initial failures.

The technical issues were fixed in a few months, but fixing the root cause -- including bringing in clear owners of the project to ensure that the process and cultural facets of devops were addressed -- was multi-faceted and spanned many more months, Dawson says.

Only then was the agency "able to properly and fully implement devops on all the planes of people, process, and tools," Dawson says.

Devops no doubt offers great promise in accelerating your software delivery cycles, but it's up to you and your team to deliver on that promise with a cohesive devops culture and sound devops practices.

 

Previous Page  1  2  3  4 

Sign up for CIO Asia eNewsletters.