It is extremely important to develop a strategic plan when creating test automation scripts. With the push towards Agile and DevOps the mindset has become to automate as much testing as possible. While that idea makes sense, it is important to determine how much test automation is needed. Part of the problem is that those who are managing Agile or DevOps teams have a really different mindset, and they really don’t understand basic testing principles. Instead of taking the time to understand, they force their understanding and push test engineering in the wrong direction.
Test Automation Planning
Test automation is certainly critical for Agile and DevOps. There are things that can and should be automated but before that is done, a few questions must be asked:
How many times are these test automation scripts going to be executed?
Is the code stable or is it going to change a lot over the next several sprints?
Has the test been executed manually and has it passed successfully?
How much time will it take to maintain the test scripts?
Do you have the right resources to build and update the test scripts?
What type of tools are you going to use?
What type of framework are you going to build?
How much does the test automation software cost?
These questions will help lay the foundation of your test automation planning. The next step will be to identify the number of test cases that need to be automated.
Testing That Should Not Be Automated
Here is a list of test cases that should not be automated:
Test cases that will only be executed a few times
Test cases that require human touch, for example review of a customer bill
Test cases that require a third party billing system such as a payment gateway that doesn’t have a test environment
Test cases were the code is unstable and is constantly changing
Test Case Quality
Test automation scripts help with providing test coverage across applications. This allows faster deployments because it doesn’t require manual testing. However, if you create hundreds of test cases and those never identify any defects, is it really worth maintaining and running those over and over again? Probably not. The test cases that are automated should be able to capture defects. Therefore, it is important that you have some really sharp manual testers who can think through the testing scenarios and identify problem areas that need attention. Without that system knowledge, you will not capture defects and it will not be a good use of your resources.
Return on Investment
It is really important to understand how much cost is involved in creating automated tests. If you create thousands of them, how much is it going to take to make changes on those when the GUI or back end systems change? If it is your department or team, as a manager or employee, you really need to be able to answer this challenging question. At every organization, I make sure that I calculate the ROI. There are a few ways to do this. First, you roughly need to know what is the cost of a member of your team or the average rate per contractor or employee. Most organizations will have that already. Second, you will need to determine how many scripts are going to be automated and how many times are they going to be executed. Third, you need to know how much time does it take to execute those test cases manually. Fourth, once the test cases are automated, you need to know how much time does each one take to execute. These items will at least give you a basic ROI. Most ROI numbers are somewhere between 6-12 months. Anything over a year, probably isn’t worth it. There are other variations that are more complex, but this will at least get you started. Knowing this information is certainly powerful, and it will help justify the expense for having automation engineers.
I cannot stress the importance of doing these basic steps. With Agile and DevOps, everything tends to be rushed. Perform this basic analysis first and spend a little time up front to yield large returns in the long run.
If you are looking to learn more information about DevOps it will take you a while to figure out what are the top 2 DevOps books on the internet today. I am not a huge fan of reading, so when I read, I really like to enjoy the books I read. In addition, you want to make sure that you are getting the right information. There are so many thoughts around DevOps, and there is quite honestly a lot of misinterpretation on the subject, whether it is at a conference, on a blog, or written in books. You want to make sure you can understand DevOps best practices so that you can effectively implement those either as a consultant or an employee for the organization you are working for. As you probably already know DevOps is one of the key cornerstones of organizations that are looking to transform their development and operations processes. Anyone you talk with today is going to mention DevOps over and over again as a part of their IT transformation strategy. You want to make sure you understand and can actively contribute to the organizational objectives. Simply put, DevOps is DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives.
Several years ago, I was at a conference, and I heard about a particular book that sounded really interesting. It isn’t your typically technology book because it is written as a fictional novel. Is is called The Phoenix Project. It really is a fantastic book and quick, easy read. The book helps you to identify with key challenges and issues that are facing most corporations from both a business and a technology perspective. Here is a high level summary of the book.
Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.
With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.
In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.
It really helps to be able to understand the challenges that Bill faces, and the steps he and others take in order to overcome the daily struggles. The organization is constantly putting out fires, so they are unable to get ahead of these issues and develop a long term strategy. I have personally been in several organizations, and part of the reason why I really enjoyed the book was that I could identify with the challenges that the characters in the book were facing. The Phoenix Project authors (Gene Kim, Kevin Behr, George Spafford) got together and wrote the book leveraging the ideas and concepts from a prior book call The Goal, which I also recommend.
The second book which I highly recommend is The DevOps Handbook. This book is more of what you would typically expect from a technology book. It helps you to gain a much better understanding of what DevOps is, and what steps you can take to implement DevOps in your organization. The authors (Gene Kim, Jez Humble, Patrick Debois, John Willis) are all industry heavyweights and really have perfected the art of DevOps and are helping others to do so. Here is a high level summary of the book.
More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.
And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.
Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.
I think a combination of both of these books will help you gain a tremendous understanding of DevOps. After the initial reads, I keep both of these books handy, and I have read them at least a few times. In addition to that, I have recommended them multiple times to everyone who is interested in learning more about this software development practice.
»DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.
»Under a DevOps model, development and operations teams are no longer “siloed.” Sometimes, these two teams are merged into a single team where the engineers work across the entire application lifecycle, from development and test to deployment to operations, and develop a range of skills not limited to a single function. and security teams may also become more tightly integrated with development and operations and throughout the application lifecycle.
»These teams use practices to automate processes that historically have been manual and slow. They use a technology stack and tooling which help them operate and evolve applications quickly and reliably. These tools also help engineers independently accomplish tasks (for example, deploying code or provisioning infrastructure) that normally would have required help from other teams, and this further increases a team’s velocity.
»2008- Patrick Debois helps plant the seeds of the DevOps movement at the Agile conference in Toronto
»2009-O’Reilly Velocity Conference—John Allspaw and Paul Hammond deliver a seminal talk known as “10+ Deploys per Day”
»2009-Debois launches the first Devopsdays event, in Ghent, Belgium
»2010-The first US Devopsdays is organized, with the help of Willis as well as other early DevOps proponents like Damon Edwards and Andrew Clay Shafer
»2011- Cameron Haight from Gartner, among others, predicts that by 2015, 20 percent of global 2000 businesses will embrace DevOps.
»2012-various Devopsdays are suddenly popping up around the world, from Bangalore to Boston.
»2013-Various DevOps books emerge: Phoenix Project (Kim), Implementing Lean Software Development” (Poppendiek), The Lean Startup (Ries), Web Operations (Allspaw), Continuous Delivery, (Humble and David Farley) and The Goal (Goldratt)
»2014-DevOps crosses into the enterprise, and established brands like Target, Nordstrom and LEGO embrace the movement. In a survey by Puppet Labs, IT Revolution Press and ThoughtWorks, 16 percent of 1,485 respondents say they are part of a DevOps effort within their organization.
In today’s IT world the need for organizations to protect sensitive information of their customers is more than ever. Governments around the world have enforced policies to protect companies against poor management of sensitive information. Compliance is one the very critical components in IT. But in IT this has a negative inference. The reason being it brings too much of documented processes involving discussions, paperwork which in turn slows down the more important work of releasing the software.
As the trend of implementing DevOps in IT organizations continue there is a need to understand how this will affect security and compliance controls. These controls are key enablers of software development activities. So it is very important for organizations to think how regulatory compliance fit in with the cultural and organizational shift which is brought by DevOps to support continuous delivery of software.
Regulatory Compliance in DevOps World
DevOps advocates delivering software in agile fashion by automating the IT delivery chain. This appears to contradict the regulatory goal of compliance to ensure that the organization isn’t opening itself up to potential vulnerabilities. Regulatory compliance are mandatory in regulated industries like Healthcare, Banking and Finance.
DevOps automation techniques for continuous delivery can appear to dissatisfy regulatory compliance. But the fact is that DevOps automation can help organizations not only to stay compliant, but actually increase their compliance levels. DevOps teams must not only include compliance activities early in the software life cycle as in the case of testing but also automate the compliance tests. Instead of leaving the security and compliance concerns for later in the release cycle it is better to deal with them early in the development life cycle. By doing so it will be easy to remove the compliance bottleneck early and bring security, quality, agility and stability into the software value chain. Automation of processes and tests will help reduce the risk of introducing security flaws due to human error.
Maintaining audit trails of software development activities is the key requirement of regulatory compliance. The continuous integration mechanism in DevOps can log and track precisely what version of each source code file contributed to which version of the software. Also with continuous deployments each build can be tagged to assure that the deployments are inspected to deny unauthorized changes moving forward. The automation testing of every build can be checked for regulatory issues. Automation of infrastructure provisioning is another area of DevOps that will help satisfy the regulatory compliance. Infrastructure provisioning scripts as verifiable and testable which can be reliable and reproduce results. All these satisfy the mandatory regulatory compliance.
By adopting the DevOps automation techniques which extend the entire software delivery pipeline, the ability to control, audit, and protect the organizational assets will only increase. This will ensure that the organizations are compliant with regulatory requirements.
Conclusion
DevOps guiding principles like automation and validation actually provide in depth audit and change information to satisfy audit and regulatory compliance needs. Another compliance concern which is governance is also addressed in DevOps by advocating processes for creating, communicating, and enforcing policies which also includes security and compliance policies across an organization. DevOps actually complements existing processes and methodologies such as ITIL & Agile which help organizations to be compliant.
Quality Assurance has always been an evolving discipline in software development. With the emerging trends in IT industry, the need to better understand, manage, and adopt the QA activities is increasing. With the onset of agile and lately DevOps, the way organizations develop software has changed, and so have the ways to enforce QA. Software development cycles have become short and quick. With this QA teams face new challenges as they work to keep pace. The advantages of overcoming the challenges include quality, optimization, process improvement and higher productivity.
Understanding QA in DevOps Landscape
DevOps advocates good principles and practices that help improve communication and collaboration between the organizational silos. This also implies to QA organization and their development counterparts. But in a DevOps scenario, the walls will be eliminated and this helps facilitate knowledge sharing, experience and specialized skills to deliver quality systems. In the era of DevOps the focus of QA teams will be more on preventing defects than finding them.
Challenges faced by QA teams
QA culture – In the context of DevOps, quality requires a change in how it is being conducted. This also implies an intense transference in the organizational culture as well. It is very important and also challenging to think of innovative ways of identifying unique techniques to test the software quickly and efficiently. This will enable to continuously ensure quality while also growing and evolving the QA services provided.
Facilitation of quality – From a DevOps perspective, QA team needs to understand the business for the system being verified. For this to happen, QA team should partner with business experts, including the product owner, to understand how the system being tested needs to function in order to support the business. QA teams will be disabled if not involved in those initial discussions. This involvement helps QA to become the facilitator of quality.
Collaboration – QA is the binding entity between development and operations. So QA team should be involved right from the early stages of development. This will enable them to collaborate to have software developed and supported more effectively. Also QA should be considered as responsibility of entire project team rather than the responsibility of dedicated QA team.
Early testing – One of the main objectives of testing in DevOps is early detection of defects in development cycle. For this to happen testing must begin very early in the cycle. QA should begin testing with whatever code is available even if the features are not complete. This requires lot of maturity in documenting self-sufficient user stories that do not depend on another for testing.
Test coverage – In DevOps there is a rush to deliver software quickly with the techniques like continuous integration and deployment. Also because of rapidly changing requirements, there is a possibility to miss testing critical functions. To overcome this challenge, a thorough and detailed traceability of requirements to test functions should be maintained.
Build verification – As DevOps encourages frequent builds, there is a higher possibility of code breaking existing features. For this reason it is not practical to have testers do these verifications manually. It is recommended to rely on automated testing for these smoke tests.
Conclusion
If the above discussed challenges are addressed then QA in DevOps can play a critical role in accelerating development and release schedules. DevOps guiding principles like test first, free communication and seamless collaboration help resolve some of the QA challenges and also enable the QA team to take their deliverables to the next level. In DevOps testing is a continuous process and supports the process of incorporating continuous feedback to enable better quality.
