Creating a Bunch of Test Automation Scripts Is a Waste of Money

It is extremely important to develop a strategic plan when creating test automation scripts.  With the push towards Agile and DevOps the mindset has become to automate as much testing as possible.  While that idea makes sense, it is important to determine how much test automation is needed.  Part of the problem is that those who are managing Agile or DevOps teams have a really different mindset, and they really don’t understand basic testing principles.  Instead of taking the time to understand, they force their understanding and push test engineering in the wrong direction.

Test Automation Planning

Test automation is certainly critical for Agile and DevOps.  There are things that can and should be automated but before that is done, a few questions must be asked:

  • How many times are these test automation scripts going to be executed?
  • Is the code stable or is it going to change a lot over the next several sprints?
  • Has the test been executed manually and has it passed successfully?
  • How much time will it take to maintain the test scripts?
  • Do you have the right resources to build and update the test scripts?
  • What type of tools are you going to use?
  • What type of framework are you going to build?
  • How much does the test automation software cost?

These questions will help lay the foundation of your test automation planning.  The next step will be to identify the number of test cases that need to be automated.

Testing That Should Not Be Automated

Here is a list of test cases that should not be automated:

  • Test cases that will only be executed a few times
  • Test cases that require human touch, for example review of a customer bill
  • Test cases that require a third party billing system such as a payment gateway that doesn’t have a test environment
  • Test cases were the code is unstable and is constantly changing

Test Case Quality

Test automation scripts help with providing test coverage across applications.  This allows faster deployments because it doesn’t require manual testing.  However, if you create hundreds of test cases and those never identify any defects, is it really worth maintaining and running those over and over again?  Probably not.  The test cases that are automated should be able to capture defects.  Therefore, it is important that you have some really sharp manual testers who can think through the testing scenarios and identify problem areas that need attention.  Without that system knowledge, you will not capture defects and it will not be a good use of your resources.

Return on Investment

It is really important to understand how much cost is involved in creating automated tests.  If you create thousands of them, how much is it going to take to make changes on those when the GUI or back end systems change?  If it is your department or team, as a manager or employee, you really need to be able to answer this challenging question.  At every organization, I make sure that I calculate the ROI.  There are a few ways to do this.  First, you roughly need to know what is the cost of a member of your team or the average rate per contractor or employee.  Most organizations will have that already.  Second, you will need to determine how many scripts are going to be automated and how many times are they going to be executed.  Third, you need to know how much time does it take to execute those test cases manually.  Fourth, once the test cases are automated, you need to know how much time does each one take to execute.  These items will at least give you a basic ROI.  Most ROI numbers are somewhere between 6-12 months.  Anything over a year, probably isn’t worth it.  There are other variations that are more complex, but this will at least get you started.  Knowing this information is certainly powerful, and it will help justify the expense for having automation engineers.

I cannot stress the importance of doing these basic steps.  With Agile and DevOps, everything tends to be rushed.  Perform this basic analysis first and spend a little time up front to yield large returns in the long run.

What are the Top 2 DevOps Books?

If you are looking to learn more information about DevOps it will take you a while to figure out what are the top 2 DevOps books  on the internet today.  I am not a huge fan of reading, so when I read, I really like to enjoy the books I read.  In addition, you want to make sure that you are getting the right information.  There are so many thoughts around DevOps, and there is quite honestly a lot of misinterpretation on the subject, whether it is at a conference, on a blog, or written in books.  You want to make sure you can understand DevOps best practices so that you can effectively implement those either as a consultant or an employee for the organization you are working for.  As you probably already know DevOps is one of the key cornerstones of organizations that are looking to transform their development and operations processes.  Anyone you talk with today is going to mention DevOps over and over again as a part of their IT transformation strategy.  You want to make sure you understand and can actively contribute to the organizational objectives.  Simply put, DevOps is DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives.

Top 2 DevOps Books

Top DevOps Book: The Phoenix Project

Several years ago, I was at a conference, and I heard about a particular book that sounded really interesting.  It isn’t your typically technology book because it is written as a fictional novel.  Is is called The Phoenix Project.  It really is a fantastic book and quick, easy read.  The book helps you to identify with key challenges and issues that are facing most corporations from both a business and a technology perspective.  Here is a high level summary of the book.

Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced. 

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited. 

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.

It really helps to be able to understand the challenges that Bill faces, and the steps he and others take in order to overcome the daily struggles.  The organization is constantly putting out fires, so they are unable to get ahead of these issues and develop a long term strategy.  I have personally been in several organizations, and part of the reason why I really enjoyed the book was that I could identify with the challenges that the characters in the book were facing.  The Phoenix Project authors (Gene Kim, Kevin Behr, George Spafford) got together and wrote the book leveraging the ideas and concepts from a prior book call The Goal, which I also recommend.

Top DevOps Book:  The DevOps Handbook

The second book which I highly recommend is The DevOps Handbook.  This book is more of what you would typically expect from a technology book.  It helps you to gain a much better understanding of what DevOps is, and what steps you can take to implement DevOps in your organization.  The authors (Gene Kim, Jez Humble, Patrick Debois, John Willis) are all industry heavyweights and really have perfected the art of DevOps and are helping others to do so.  Here is a high level summary of the book.

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.

And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

I think a combination of both of these books will help you gain a tremendous understanding of DevOps.  After the initial reads, I keep both of these books handy, and I have read them at least a few times.  In addition to that, I have recommended them multiple times to everyone who is interested in learning more about this software development practice.

DevOps 101: Learn all about DevOps

What is DevOps?

DevOps»DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.

»Under a DevOps model, development and operations teams are no longer “siloed.” Sometimes, these two teams are merged into a single team where the engineers work across the entire application lifecycle, from development and test to deployment to operations, and develop a range of skills not limited to a single function.  and security teams may also become more tightly integrated with development and operations and throughout the application lifecycle.

»These teams use practices to automate processes that historically have been manual and slow. They use a technology stack and tooling which help them operate and evolve applications quickly and reliably. These tools also help engineers independently accomplish tasks (for example, deploying code or provisioning infrastructure) that normally would have required help from other teams, and this further increases a team’s velocity.

DevOps Benefits

Here are some DevOps benefits:

  • Speed: Move faster as a team so there will  be more product produced
  • Quality: Develop higher quality products with faster feedback loops resulting in continuous quality deliveries
  • Collaboration: Smaller teams work more efficiently together and eliminate organizational silos
  • Scale: Manage development and infrastructure leveraging automation to accelerate productivity
  • Security: Using a DevOps model companies can ensure security compliance

Developing a Quality DevOps Mindset

»Customer is #1

»Frequent delivery of a working software product

»Develop a quality product over production delivery

»Keep technical debt at a minimum

»Constant communication with all team members

»Keep things simple as possible

»Hold meaningful and actionable retrospective sessions

»Automate as much as possible

DevOps Myths

»DevOps will fix everything

»DevOps is for Developers and Operations only

»DevOps only involves automating processes

»DevOps only uses “Cloud

»DevOps only uses new languages and technologies

»DevOps is only for web based systems

»DevOps is only used by small companies

»DevOps is not necessary for my organization

DevOps History

»2008- Patrick Debois helps plant the seeds of the DevOps movement at the Agile conference in Toronto

»2009-O’Reilly Velocity Conference—John Allspaw and Paul Hammond deliver a seminal talk known as “10+ Deploys per Day”

»2009-Debois launches the first Devopsdays event, in Ghent, Belgium

»2010-The first US Devopsdays is organized, with the help of Willis as well as other early DevOps proponents like Damon Edwards and Andrew Clay Shafer

»2011- Cameron Haight from Gartner, among others, predicts that by 2015, 20 percent of global 2000 businesses will embrace DevOps.

»2012-various Devopsdays are suddenly popping up around the world, from Bangalore to Boston.

»2013-Various DevOps books emerge: Phoenix Project (Kim), Implementing Lean Software Development” (Poppendiek), The Lean Startup (Ries), Web Operations (Allspaw), Continuous Delivery, (Humble and David Farley) and The Goal (Goldratt)

»2014-DevOps crosses into the enterprise, and established brands like Target, Nordstrom and LEGO embrace the movement. In a survey by Puppet Labs, IT Revolution Press and ThoughtWorks, 16 percent of 1,485 respondents say they are part of a DevOps effort within their organization.

2016-DevOps is now mainstream

DevOps Goals

»Build repeatable processes and high quality software

»Automate everything

»Definition of Done

»Reduce Risk

»Collaborative Ownership-Everyone is responsible

»Constant communication

»Continuous Improvements

I hope this information has been helpful. I also have a DevOps 101 video. and a DevOps 201 video.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

QA Revolution YouTube Channel

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Characteristics of Test Data Management

  • Introduction

Test data plays an important role in successful completion of test. Test teams not only have to follow exact test methodologies, but also ensure accuracy of data to correctly reflect production situations, both functionally and technically.

Test data scope starts with receiving the request for test data and criteria from the testing team and ends with delivering the data. This includes setting up the data environment, loading test data from production environment or creation of test data (these vary from project to project). Test data is a critical need for Functional, Integration, System, Acceptance, Automation and Performance Testing. This depends on the agreement among the various stake holders.

A well-defined Test data management strategy can rapidly reduce inefficiencies, help extract greater value from expensive data and make validated test data available in an organized, secured, consistent and controlled manner.

The exact nature or representation of the production data needs to be understood through data profiling and discussions with business analysts. This will help understand what makes the data valuable.

Some key pieces of information shall be focused during this process:

  • Domain values: The full range of valid and meaningful values for a data field
  • Data ranges and limits: Especially those that define our equivalence classes
  • Data relationships: Data characteristics including cross- system data mappings and sources for derived or calculated data
  • Upstream and downstream: Data dependencies from upstream and downstream systems

The test data management team should understand how the above mentioned vital features are consumed by the business users.

The type of testing also impacts the test data requirement management. For example, an automation testing scenario requires highly stable, predictable data sets whereas manual testing can afford some variability. Performance tests usually require test data either represented or sampled from the production environment to replicate the real production scenario.

The following are the important activities that are performed as part of Test Data Management:

  • Initial setup of test data: This is a one time job which requires initial setup and synchronization of test data.
  • Service projects for test data requirements: Provide test data to projects based on the requests received. Projects raising data requests may be either new project which require the data to be created from scratch or can be maintenance projects where the data needs to be refreshed.
  • Continuous support to projects for test data requirements: Maintenance of test data based on requests sent by the projects. Like simple data creation request for change in data requirements, requests related to rectifying problems related to test data delivered.
  • Maintenance of data: Scheduled maintenance of test data beds in defined frequencies (weekly, monthly, quarterly or annually)

 Test Data Management Characteristics

A test data management elements are helpful not only to organize information about test data, but also to maintain this information over time.  The following data characteristics or attributes are critical to build the framework:

  • Data classification
  • Data sources
  • Data selection

Data Classification

Data classification has to be considered as an important parameter for an effective test data management.  Data classification includes the following:

  • Environmental data
  • Baseline data
  • Input data

Environmental data defines the application operational environment and is a foundational component of the test effort since it establishes our execution context. Environmental data includes:

  • System configuration
  • User authorization, authentication, and credentials
  • Configuration options

Baseline data has two fundamental purposes – to establish a meaningful starting point for testing and to establish a set of expected results. The initial baseline is set from test case pre-requisites from reliable data which helps in an ideal environment to attain the expected results. This shall help the test team reduce the efforts through automation for result evaluation process against an expected result baseline.

Input data is the data entered into the system under test to evaluate how it responds to the provided input. Observed behaviour establishes the actual results which must then be compared to expected results to determine the correctness of the behavior. Input data is typically a component of the test case itself.

Data Sources

The following are the major sources of test data:

  • Production data
  • Derivatives of production data
  • Simulated data

Production data is rarely used for testing given the risk of data security concerns and regulatory compliance requirements.   In some scenarios when it is inevitable that production data must be used for the test management the test teams should be aware of the sensitivity of the data with which they are working.

Derivatives of production data sets help to maintain the production like characteristics. A proper sanitization needs to be applied on such data to minimize the risk of security breaches.

Simulated data is useful when there is limited or no production data available. The simulated data fits to unit testing where less effort is required to create the test data.

Data Selection

All the possible scenarios need to be considered in the preparation of the test data.   Any scenarios not considered in data preparation will impact the quality of the test data used for testing.  Sample scenarios that will affect the data selection criteria are:

  • Positive and negative testing scenarios
  • Possible overlaps and redundancies
  • Statistical characteristics of the data
  • Default conditions
  • Cross-project dependencies

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Regulatory Compliance & DevOps


In today’s IT world the need for organizations to protect sensitive information of their customers is more than ever. Governments around the world have enforced policies to protect companies against poor management of sensitive information. Compliance is one the very critical components in IT. But in IT this has a negative inference. The reason being it brings too much of documented processes involving discussions, paperwork which in turn slows down the more important work of releasing the software.

As the trend of implementing DevOps in IT organizations continue there is a need to understand how this will affect security and compliance controls. These controls are key enablers of software development activities. So it is very important for organizations to think how regulatory compliance fit in with the cultural and organizational shift which is brought by DevOps to support continuous delivery of software.

Regulatory Compliance in DevOps World

DevOps advocates delivering software in agile fashion by automating the IT delivery chain. This appears to contradict the regulatory goal of compliance to ensure that the organization isn’t opening itself up to potential vulnerabilities. Regulatory compliance are mandatory in regulated industries like Healthcare, Banking and Finance.

DevOps automation techniques for continuous delivery can appear to dissatisfy regulatory compliance. But the fact is that DevOps automation can help organizations not only to stay compliant, but actually increase their compliance levels. DevOps teams must not only include compliance activities early in the software life cycle as in the case of testing but also automate the compliance tests. Instead of leaving the security and compliance concerns for later in the release cycle it is better to deal with them early in the development life cycle. By doing so it will be easy to remove the compliance bottleneck early and bring security, quality, agility and stability into the software value chain. Automation of processes and tests will help reduce the risk of introducing security flaws due to human error.

Maintaining audit trails of software development activities is the key requirement of regulatory compliance. The continuous integration mechanism in DevOps can log and track precisely what version of each source code file contributed to which version of the software. Also with continuous deployments each build can be tagged to assure that the deployments are inspected to deny unauthorized changes moving forward. The automation testing of every build can be checked for regulatory issues. Automation of infrastructure provisioning is another area of DevOps that will help satisfy the regulatory compliance. Infrastructure provisioning scripts as verifiable and testable which can be reliable and reproduce results. All these satisfy the mandatory regulatory compliance.

By adopting the DevOps automation techniques which extend the entire software delivery pipeline, the ability to control, audit, and protect the organizational assets will only increase. This will ensure that the organizations are compliant with regulatory requirements.


DevOps guiding principles like automation and validation actually provide in depth audit and change information to satisfy audit and regulatory compliance needs. Another compliance concern which is governance is also addressed in DevOps by advocating processes for creating, communicating, and enforcing policies which also includes security and compliance policies across an organization. DevOps actually complements existing processes and methodologies such as ITIL & Agile which help organizations to be compliant.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.