Creating a Bunch of Test Automation Scripts Is a Waste of Money

It is extremely important to develop a strategic plan when creating test automation scripts.  With the push towards Agile and DevOps the mindset has become to automate as much testing as possible.  While that idea makes sense, it is important to determine how much test automation is needed.  Part of the problem is that those who are managing Agile or DevOps teams have a really different mindset, and they really don’t understand basic testing principles.  Instead of taking the time to understand, they force their understanding and push test engineering in the wrong direction.

Test Automation Planning

Test automation is certainly critical for Agile and DevOps.  There are things that can and should be automated but before that is done, a few questions must be asked:

  • How many times are these test automation scripts going to be executed?
  • Is the code stable or is it going to change a lot over the next several sprints?
  • Has the test been executed manually and has it passed successfully?
  • How much time will it take to maintain the test scripts?
  • Do you have the right resources to build and update the test scripts?
  • What type of tools are you going to use?
  • What type of framework are you going to build?
  • How much does the test automation software cost?

These questions will help lay the foundation of your test automation planning.  The next step will be to identify the number of test cases that need to be automated.

Testing That Should Not Be Automated

Here is a list of test cases that should not be automated:

  • Test cases that will only be executed a few times
  • Test cases that require human touch, for example review of a customer bill
  • Test cases that require a third party billing system such as a payment gateway that doesn’t have a test environment
  • Test cases were the code is unstable and is constantly changing

Test Case Quality

Test automation scripts help with providing test coverage across applications.  This allows faster deployments because it doesn’t require manual testing.  However, if you create hundreds of test cases and those never identify any defects, is it really worth maintaining and running those over and over again?  Probably not.  The test cases that are automated should be able to capture defects.  Therefore, it is important that you have some really sharp manual testers who can think through the testing scenarios and identify problem areas that need attention.  Without that system knowledge, you will not capture defects and it will not be a good use of your resources.

Return on Investment

It is really important to understand how much cost is involved in creating automated tests.  If you create thousands of them, how much is it going to take to make changes on those when the GUI or back end systems change?  If it is your department or team, as a manager or employee, you really need to be able to answer this challenging question.  At every organization, I make sure that I calculate the ROI.  There are a few ways to do this.  First, you roughly need to know what is the cost of a member of your team or the average rate per contractor or employee.  Most organizations will have that already.  Second, you will need to determine how many scripts are going to be automated and how many times are they going to be executed.  Third, you need to know how much time does it take to execute those test cases manually.  Fourth, once the test cases are automated, you need to know how much time does each one take to execute.  These items will at least give you a basic ROI.  Most ROI numbers are somewhere between 6-12 months.  Anything over a year, probably isn’t worth it.  There are other variations that are more complex, but this will at least get you started.  Knowing this information is certainly powerful, and it will help justify the expense for having automation engineers.

I cannot stress the importance of doing these basic steps.  With Agile and DevOps, everything tends to be rushed.  Perform this basic analysis first and spend a little time up front to yield large returns in the long run.

Manual Testing Isn’t Dead!

If you are a manual software tester, you have probably read or been informed by your boss that manual testing is slowly dying and that you need to upgrade your skills and become an automation engineer.  I don’t think this can be further from the truth.  While it is true that there is value in upgrading your skills and learning test automation, manual testing will always be a valuable skillset.

Why Manual Testing Isn’t Dead

  • Manual testing can uncover some significant defects through exploratory or adhoc testing
  • Test cases are typically built by the manual tester which requires analysis and a deep understanding of the application being tested
  • If test cases are automated, they will typically first be created by the manual tester and then automated
  • There are some test cases can’t be automated.  Test cases that require analysis of bills for example shouldn’t be automated
  • Some test cases shouldn’t be automated because it doesn’t provide a good return on investment
  • A manual tester sees trends and typically knows where the defects are found
  • When schedules are compressed, the manual tester brings value in prioritizing what is tested

Recently with things such as machine learning and AI tends to surface the question again.  While machine learning and AI can be valuable additions, I don’t see the value of the manual tester decreasing anytime soon.  Learn why manual testing isn’t dead.  Many software testing organizations have some manual testing rockstars and they wouldn’t trade those valuable resources for anything.

Software Testing | Agile | DevOps | Quality Engineering

Welcome to the QA Revolution!

What is the QA Revolution?

software testingSo you are probably curious about the name of this website and want to learn more about it.  I have been building this website for many years.  My passion is software testing and I really enjoy learning as software testing is changing every year.  I enjoy sharing my knowledge and passion with other people.  I am hoping this website will be helpful for you and that you will learn something when you stop for a visit.

Many years ago, while I was completing my MBA, I began to learn the importance of marketing and how it could help transform the current testing organization that I was leading.  I met with the team and discussed the idea and wanted to learn more about why we were struggling and what we needed to do to become a critical pillar within the IT organization.  We talked about some of the mistakes we had made in missing some defects, but I also encouraged them that we needed to build a brand and market it outside our team.  Everybody embraced it, and we came up with a team name and called ourselves The QA Revolution.  I wanted the team to be able to hold their heads up high and be proud of the work that they had done.  We built a quality brand and literally started QA Revolution!  We created posters, created a logo, had signature lines for our email and developed websites that would help to build our brand.  This was a tremendous hit, and because of that, suddenly being on the QA team was cool, and others wanted to come join the team.  And believe it or not, they did.  I am talking developers and engineers..the cool kids wanted to join the cooler organization.

I tell the story in order to build a point, that building a brand is important.  It helps to define who you are and where you are going.  It establishes a vision with a mission.  These days people are looking for something to believe in that they can fully embrace.  Those memories will live on forever, and I am looking forward to be able to reproduce that again and again.

Agile Transformation

The Agile methodology has turned traditional software testing organizations on its head.  We have real world experience transforming multiple industries and successfully implementing best agile testing practices.  Agile done well can have a tremendous impact.  The agile product methodology is the one that I have successfully used at one of my most recent positions.  It is important to leverage best practices when using agile.  Developing software in short agile sprints with high quality production delivery can put your organization far ahead of your competition.  There are many learning opportunities in reaching agile nirvana.  It is a process and more of a marathon than a sprint.  Some challenges I have learned along the way is there are agile mistakes along the way.  It is fine to make mistakes since no organization is perfect.  The important thing is to learn from your mistakes.  From a testing perspective, things like end to end testing are often forgotten because they take too long.  The result of that is poor quality, which is something that your business organization cannot afford.

DevOps

DevOpsDevOps has blurred the lines between development and testing.  It has accelerated the pace of automated code deployments and given opportunities to integrate automated testing into the process.  DevOps is simply the processes of collaboration between development and the operations team.  While the goal is to make processes more efficient automating everything may not be the best answer.  It has to be determined within each organization.  The ultimate goal is to help your business win.  If you have built a killer DevOps team, but don’t set your business up for success, you will have missed the goal entirely.  I believe the DevOps practices are here to stay, but I have used these same techniques for many years before the word was invented.  DevOps enables testers to spend more time testing and less time waiting for test environments to be setup.  If you are looking for more information on DevOps, I have published an article that highly recommends reading The Phoenix Project and The DevOps Handbook which are both written by the DevOps Rockstar Gene Kim.  I don’t like to read but I really enjoyed these two books.  It is critical in invest time in learning DevOps.  There are DevOps challenges that will happen as you standup the organization, so you need to be realistic and allocate enough time and resources to do it properly.  Once you reach a steady state, then you can focus on continuous process improvements which will really get you to the next level and provide some significant DevOps benefits.  DevOps is really very complementary enabling software testing transformation.

Test Automation

Test Automation has gone mainstream and all software testing organizations must do it to keep up with business demand.  It requires extreme technical expertise and multiple toolsets are needed.  Test automation is my passion and I have built multiple organizations and driven automation up to 95% automated.  I believe all software testing organizations need a strong test automation practice.  I have used both Selenium and Unified Functional Test to help deliver strong value in automating pretty much everything from mainframe to mid-range to web front ends. There needs to be a strong focus on automating what makes sense, and what will provide a good return on your investment, since it can be very expensive to maintain.  Creation of an effective test automation framework is critical so that it can handle most of the workload.  The beauty of an automation framework is that you can leverage it, instead of building thousands of scripts that will have to be individually coded.

Quality Engineering

Over the years, software testing has evolved.  It has grown from some fantastic software testers to quality engineers.  Manual testing has evolved for sure over the last 20 years into something that is more sophisticated.  Sure, there will be manual testing forever, because everything does not need to be automated, but the days of having only manual testers are long gone.  In order to keep up with the demand, you have to have highly skilled individuals with development type skills.  Notice I said development type skills, not developers, because I firmly believe that a great developer may not transform into a great tester.  The key is a mindset.  Are you curious?  Do you ask a lot of questions? Are you analytical?  Do you like numbers?  Can you see typos and other errors better than anyone else?  Will you ask the difficult questions and challenge the status quo?  These are some of the questions I ask when I talk to potential candidates.  While having a stacked resume with skills, education, and certifications are truly impressive, they will not answer the questions above.  A quality engineer needs to be able to seek, probe and find those edge cases where nobody else dares to venture.  Quality Engineering has somewhat merged with DevOps, so those lines are starting to cross.  Those that can think, code, and solve complex problems are the quality engineers who will have long careers in this field.  Are you one of those?  Are you looking to build your career into a quality engineering expert?

I have built many years of knowledge on The QA Revolution website.  I encourage you to look around and learn more.  I am constantly adding new content that is relevant to the rapidly changing technology landscape that we currently live in.

 

What are the Top 2 DevOps Books?

If you are looking to learn more information about DevOps it will take you a while to figure out what are the top 2 DevOps books  on the internet today.  I am not a huge fan of reading, so when I read, I really like to enjoy the books I read.  In addition, you want to make sure that you are getting the right information.  There are so many thoughts around DevOps, and there is quite honestly a lot of misinterpretation on the subject, whether it is at a conference, on a blog, or written in books.  You want to make sure you can understand DevOps best practices so that you can effectively implement those either as a consultant or an employee for the organization you are working for.  As you probably already know DevOps is one of the key cornerstones of organizations that are looking to transform their development and operations processes.  Anyone you talk with today is going to mention DevOps over and over again as a part of their IT transformation strategy.  You want to make sure you understand and can actively contribute to the organizational objectives.  Simply put, DevOps is DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives.

Top 2 DevOps Books

Top DevOps Book: The Phoenix Project

Several years ago, I was at a conference, and I heard about a particular book that sounded really interesting.  It isn’t your typically technology book because it is written as a fictional novel.  Is is called The Phoenix Project.  It really is a fantastic book and quick, easy read.  The book helps you to identify with key challenges and issues that are facing most corporations from both a business and a technology perspective.  Here is a high level summary of the book.

Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced. 

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited. 

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.

It really helps to be able to understand the challenges that Bill faces, and the steps he and others take in order to overcome the daily struggles.  The organization is constantly putting out fires, so they are unable to get ahead of these issues and develop a long term strategy.  I have personally been in several organizations, and part of the reason why I really enjoyed the book was that I could identify with the challenges that the characters in the book were facing.  The Phoenix Project authors (Gene Kim, Kevin Behr, George Spafford) got together and wrote the book leveraging the ideas and concepts from a prior book call The Goal, which I also recommend.

Top DevOps Book:  The DevOps Handbook

The second book which I highly recommend is The DevOps Handbook.  This book is more of what you would typically expect from a technology book.  It helps you to gain a much better understanding of what DevOps is, and what steps you can take to implement DevOps in your organization.  The authors (Gene Kim, Jez Humble, Patrick Debois, John Willis) are all industry heavyweights and really have perfected the art of DevOps and are helping others to do so.  Here is a high level summary of the book.

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.

And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

I think a combination of both of these books will help you gain a tremendous understanding of DevOps.  After the initial reads, I keep both of these books handy, and I have read them at least a few times.  In addition to that, I have recommended them multiple times to everyone who is interested in learning more about this software development practice.

Career Tip#1:Build Your LinkedIn Network

Build your LinkedIn Network

The most important thing you can do your your career is to Build Your LinkedIn Network.  It is getting harder to find your next opportunity.  I have been focusing on building my LinkedIn network and there are some helpful hints that I have learned along the way.  Here they are:

  • Connect with people at your current job.  Many years before LinkedIn, you had to keep up your previous co-workers through email or phone.  LinkedIn has made it a lot easier.  I make sure that I periodically add coworkers.  It helps to keep things current.  It is really important to add those individuals who you work with and trust because they know you the best.
  • Connect with others who are industry experts.  It is important to connect with others who are influential within your industry.  They will help you learn more about your profession, and they might be able to help you land your next dream job.
  • Limit your connections within LinkedIn.  Currently there is a restriction of connecting with 30K members.  Once you hit that limit, you will not be able to add any additional people within your network.  This is somewhat limiting and many within the LinkedIn network are requesting this restriction is lifted.  Once you have hit 30K, people will need to follow you. I made the mistake early to follow everyone and then I saw the LinkedIn limit.  I am more cautious now, so I make sure that those who I connect with are relevant.  I currently have over 24K connections.
  • Follow LinkedIn Influential People: There are several people such as Oleg, Brigette, and Cory are really helping others in any way they possibly can.  I have seen many instances, where their influence has helped to get people jobs.  It is amazing to see the positive power and the influence that those are providing.  I have followed then and plan to stay connected with them and I know I will learn a lot from them.

Hopefully this information has been helpful and will help you build your LinkedIn network.  If you are looking for additional career tips please read 5 Steps to Increase Your Career Opportunities.