Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Are you over-testing your software?

Matthew Heusser | July 8, 2015
Release candidate testing takes too long.

Release candidate testing takes too long. 

For many agile teams, this is the single biggest challenge. Legacy applications start with a test window longer than the sprint. This happens over and over again with clients and colleagues working on large, integrated websites and applications. 

But what if you didn't need a human to look at a specific, numbered build to manage risk before deploy? Instead, what if a bot told the team the build was ready, and all someone had to do was click the deploy button? 

Getting there would take some infrastructure and discipline. It might not be possible for you, but there are organizations that do this every day. 

That is exactly what the technical staff does at Etsy.com, a sort of online mall for crafters that had over $1 billion in sales in 2013. Etsy had about a hundred programmers on staff in 2011. It has grown quite a bit since. New programmers at Etsy go through a process on the first day to learn the pipeline, which includes making changes, running the checks and a production deploy. Etsy isn't alone; many companies pursue a model of continuous delivery. 

The secrets to eliminating regression testing 

Companies that have eliminated or at least dramatically reduced release candidate testing have a few things in common: 

Test tooling. Many tools exist that can exercise the software, running through basic flows and looking for key conditions. This can range from unit tests or testing a Web API all the way to end-to-end exercising of the GUI. One thing to look out for: Over time, end-to-end tests tend to take so long as to become unwieldy. Picking the right automated checks to put in place is critical; the goal is to have just enough to reduce risk, run fast and diminish the maintenance burden that comes with keeping the checks up to date. 

Hook tools into the build system. Waiting for a build and then running checks by hand at a desk adds wait-time to the process. Get this to happen automatically with every build, ideally an automated checkout/build/deploy/test/promote staging pipeline. 

Radically reduce regression error. Continuous integration that find continuous problems will lead to continuous fixing and retesting. In order for these strategies to work, the code that comes out of continuous integration needs to fall back or regress much less often than industry standard. With excellent release candidate testing in place, it doesn't. Too often the safety net enables bad work. Remember, eliminating release candidate testing means improving development practices. 

Develop separately deployable components. It's true: Programmers at Etsy deploy to production on day one. The code they write, however, is a very simple change to add their name and image to the About Us > Our Team page. That's it. The change doesn't touch the database, the web services, the style sheets, any code libraries or production code; it's limited to a regular HTML file and an image. The programmer gets specific directions, so the worst that can happen is that the page looks wrong for a few minutes. 

 

1  2  3  Next Page 

Sign up for CIO Asia eNewsletters.