Manual Testing Isn’t Dead!

If you are a manual software tester, you have probably read or been informed by your boss that manual testing is slowly dying and that you need to upgrade your skills and become an automation engineer.  I don’t think this can be further from the truth.  While it is true that there is value in upgrading your skills and learning test automation, manual testing will always be a valuable skillset.

Why Manual Testing Isn’t Dead

  • Manual testing can uncover some significant defects through exploratory or adhoc testing
  • Test cases are typically built by the manual tester which requires analysis and a deep understanding of the application being tested
  • If test cases are automated, they will typically first be created by the manual tester and then automated
  • There are some test cases can’t be automated.  Test cases that require analysis of bills for example shouldn’t be automated
  • Some test cases shouldn’t be automated because it doesn’t provide a good return on investment
  • A manual tester sees trends and typically knows where the defects are found
  • When schedules are compressed, the manual tester brings value in prioritizing what is tested

Recently with things such as machine learning and AI tends to surface the question again.  While machine learning and AI can be valuable additions, I don’t see the value of the manual tester decreasing anytime soon.  Learn why manual testing isn’t dead.  Many software testing organizations have some manual testing rockstars and they wouldn’t trade those valuable resources for anything.

Agile Testing: Don’t Forget the Negative Test Scenarios

negative test scenarios

With software testing, and agile testing in particular, it is critical not to forget about the negative test scenarios.  Agile forces teams to move a break neck speeds and often we are lucky to get the happy path scenarios tested, so you are probably thinking how in the world can we also have time to cover the negative test scenarios.  Unfortunately, when there are production quality issues, the root of most of those defects is that the test scenario was not tested.  Planning in agile, contrary to what many may think is that test planning is a critical piece of the overall process of quality.

I have been running an agile team for over a year now, and we have done a great job at covering happy path scenarios, or those items which we identified which need to be tested.  However, we have done a very poor job of identifying and documenting negative test scenarios.  For our product, we manage electronic file transfers between our customers and other internal systems.  With this one scenario, two different files were received at the same time, and the second file ended up overwriting the first file.  While we were able to reprocess the overwritten files, it was a painful lesson because we did not factor that negative test scenario in our testing.

While it is impossible to account for all the different negative test scenarios, it is possible to document those which might be most probable and automate as much of those scenarios that you can, so that you can speed up the testing process.  In the event that defects are discovered like it happened on my team, it provides a great learning opportunity to grow and raise awareness which will help identify potential missed opportunities.  I would venture to guess that most agile testing covers 90 to 95 percent of happy path scenarios.  If we spend more time and identify 5 to 10 percent more negative test scenarios, then I believe our overall quality will increase.

Why is End to End Testing is Critical in Agile?

end to end testing I am a firm believer that end to end testing is critical in Agile.  I have personally experienced countless instances of where production issues could have been avoided if a few end to end tests would have been executed.  Most of my professional career has been devoted to software testing and end to end best practices so I have a high degree of understanding of the testing process.  Testing is a critical piece of the agile process but it is often misunderstood.

 

 

 

Here are some items that will help you determine if end to end testing is needed:

Dependencies: For the code change that is being made, is it isolated to a specific product or does the change impact another product or service?  If the answer is no, then no end to end testing is needed.  If the answer is yes, then end to end testing is required.  This is the #1 factor when determining if end to end testing is required and it is often the primary reason why issues occur in production.  It is important that the agile teams communicate both internally and externally.  Often an agile team will make a change and not believe there is an impact to another product or service but the impact occurs because certain misunderstandings or incorrect assumptions are made.

Critical Business Workflows:  If a code change is made within a critical business workflow, then it is always a good idea to perform end to end testing to ensure no adverse impacts occur.  Often running a few end to end tests will prevent issues from occurring in production.

Billing/Payments: It is critical to run end to end tests anytime there is a change to billing or payments.  Billing and payments can impact all businesses and it is important to spend the extra time performing end to end tests.  I have personally experienced many issues with companies due to billing and payment related issues.

Customer Impacting:  If the change has the potential to impact the customer, then end to end testing should be completed.

This end to end testing framework is not exhaustive, but it should cover the critical areas where performance testing is required.  Always if you think there is a remote potential it is a good idea to run end to end testing.  Most companies have test automation that will cover the critical business scenarios so it is always a good idea to run those automated regression tests to ensure you are covered.  Many in agile believe that end to end tests are a waste of time and not needed but I have experienced too many issues.  I also am a firm believer that if then agile team who has made the change has access to either an upstream or downstream system, then they should be able to perform the testing so they are not dependent on another team to perform the validation.  I believe that as long as you are running end to end tests that cover the code change, you don’t need to run hundreds of tests to ensure that things are working.  Run enough end to end tests but don’t run more that is needed.

 

 

Role of an Agile QA Engineer

Agile QA EngineerAgile Software Testing has literally transformed the role of an Agile QA Engineer.  There are several key differences which software testing organizations will need to adjust in order to effectively operate with Agile.

Documentation:  It isn’t practical within Agile for the Agile QA Engineer to have lots of documentation.  Don’t get me wrong, there does need to be sufficient documentation for the QA Engineers to do their job well. If a developer tries to convince you that the Agile Manifesto doesn’t require documentation, Houston we have a problem!

  • In terms of requirements, there needs to be enough documentation that the Agile QA Engineer knows how the system is supposed to operate.  In my team, we currently use a 1 page document that will provide that information for each story.  It really does help the agile testers know what is expected so they can adequately plan.
  • For the Software Test Plan, it isn’t necessary to go through all the hoopla of creating a massive plan that nobody will read or follow.  A one page document will be sufficient.  I encourage the creation of the one page plan because it forces the agile tester to think about what testing is needed.
  • For the testing sign off a simple email will suffice.

Test Automation: One of the hottest topics naturally is test automation, especially within Agile software testing.  Test automation without a doubt is critical.  However, there needs to be a strategy that is going to work.  In previous organizations, I have built world class automation teams.  We had thousands of automated test cases within our regression suites.  Were all those tests necessary, probably not.  Was it expensive to maintain, absolutely!  I believe having a centralized automation team that can build and execute the automation tests is most effective.  The team prioritizes the work based upon the needs of the various agile teams.  I have found this to work most effectively.  I usually recommend that the agile testers execute the tests within the current sprint and once those have passed, they can hand that off to the agile team to automate and put in the regression suite.  Having an Agile QA Engineer do both the functional and automated tests within a given sprint is extremely challenging and often the functional testing gets done and the automation gets put on the back burner.

Performance Testing: Another challenge is system performance.  If automation takes a back seat, then performance testing is usually behind that.  In order to deal with that challenge, having a centralized performance team will help.  Usually 2-3 performance engineers will be sufficient.  I encourage a very close look at performance testing within the Agile context.  If you don’t need it, don’t do it, if you need it, you better do it.

UI versus Web Service:  Agile software testing requires some fundamental changes in terms of how things get tested.  More and more agile software testing is taking a closer look at web service testing.  It is a critical component that can find a lot of defects.  The need for agile teams to focus on spending more time in this area is due to the amount of applications that are using API’s to transfer information.  It is a lot quicker to run hundreds of transactions through an API versus through the UI.  Agile software testing teams who are able to make this transition are simply more effective.

Let’s face it, transforming your software testing organization is going to be very challenging.  Agile software testing is extremely challenging.  The key is to have a strong team who is flexible and is adaptable to constant change.  The teams that can do that most effectively will be successful.

Role of Agile QA Manager

QA Agile Manager

If you are an Agile QA Manager your role has probably changed a good bit from the more traditional Waterfall methodology.  It certainly requires a bit adjustment, but if you are able to make the transition, it will certainly help ensure your organization delivers high quality software.  Chances are your involvement will change and it will require a different mindset and approach.  Here are a few items that will make an Agile QA Manager successful:

  1. Documentation: Certainly this is a huge change from waterfall.  In typical waterfall the documentation is a lot heavier and that includes requirements, test plans, and design documents.  Documentation is important and contrary to what you might hear, it is still an important aspect of Agile.  I still encourage my team to create a small requirement document for each story and that really helps the QA testers.  For QA, I encourage creating a 1 page QA test plan and that helps development and product owners understand what is tested and ensures everyone is on the same page.  In terms of QA sign off, I simply require an email stating that QA has signed of on the story and release.  As an Agile QA Manager, I encourage you to keep things simple and lean especially around the documentation.
  2. Collaboration: Collaboration within Agile is extremely important and it is one of the most important elements of having a high quality product.  It is important for the QA tester to work hand in hand with the BA, Developers, and other QA resources.  Collaboration helps to get the best information and gain a solid understanding of the product in order to be able to test the application more effectively.  Where possible, I encourage the QA resources to sit with the other members of the Agile team.
  3. Automation:  It is important for automation to occur, but it needs to be done efficiently.  Maintaining automation test scripts is expensive so it is important for the Agile QA Manager to take a very close look at the amount of automation in place.  I have found that the automation is most effectively done by dedicated automation resources and I have set them up as a separate team.  Many teams have the QA engineer perform both functional and automated testing, but I have found that when push comes to shove, the automation takes a back seat and isn’t a priority.
  4. Performance:  Performance testing is important and needed.  There might be applications that need more performance testing than others, so a one size fits all model doesn’t necessarily work.  This is where the experience of the Agile QA Manager comes into play, so if you think it is needed, then certainly the effort must be there.  I have also setup a small team of performance engineers that sit outside the agile team and that has worked well for me.
  5. Level of Involvement: It is important to know what the QA engineers are doing and what is being tested however sometimes it is important to take a back seat not necessary get involved in the nuts and bolts of the daily activities.  Chances are high that you will get involved when there are escalations and issues.
  6. Champion of Quality: While it is true within Agile that Quality is the teams responsibility, you need to ensure that across the organization, you are personally involved in helping to build quality.  Things like documentation and testing are important and should be consistent across all the agile teams.  The better information your team has, the chances are higher they will find the issues before they hit production.
  7. Metrics:  As an Agile QA Manager, metrics are your best friend.  It will help you pinpoint when there are issues and help prevent production disasters.  I encourage you to place a strong emphasis on metrics so you base your decisions of quality versus gut feel.

I hope this information has been helpful and I encourage you to closely take a look at all of these items so you can become a more effective Agile QA Manager.

If you are looking for more additional information on Agile roles you can find additional information here:

Role of an Agile Development Manager

Role of an Agile QA Engineer

Role of an Agile Business Analyst

Agile Software Testing Transformation

Agile Testing TransformationAgile Software Testing has literally transformed how organizations test software.  There are several key differences which software testing organizations will need to adjust in order to effectively operate with Agile.

Documentation:  It isn’t practical within Agile to have lots of documentation.  Don’t get me wrong, there does need to be sufficient documentation for the agile software testers to do their job well. If a developer tries to convince you that the Agile Manifesto doesn’t require documentation, Houston we have a problem!

  • In terms of requirements, there needs to be enough documentation that the tester knows how the system is supposed to operate.  In my team, we currently use a 1 page document that will provide that information for each story.  It really does help the agile testers know what is expected so they can adequately plan.
  • For the Software Test Plan, it isn’t necessary to go through all the hoopla of creating a massive plan that nobody will read or follow.  A one page document will be sufficient.  I encourage the creation of the one page plan because it forces the agile tester to think about what testing is needed.
  • For the testing signoff a simple email will suffice.

Automation: One of the hottest topics naturally is test automation, especially within Agile software testing.  Test automation without a doubt is critical.  However, there needs to be a strategy that is going to work.  In previous organizations, I have built world class automation teams.  We had thousands of automated test cases within our regression suites.  Were all those tests necessary, probably not.  Was it expensive to maintain, absolutely!  I believe having a centralized automation team that can build and execute the automation tests is most effective.  The team prioritizes the work based upon the needs of the various agile teams.  I have found this to work most effectively.  I usually recommend that the agile testers execute the tests within the current sprint and once those have passed, they can hand that off to the agile team to automate and put in the regression suite.  Having a tester do both the functional and automated tests within a given sprint is extremely challenging and often the functional testing gets done and the automation gets put on the back burner.

Performance: Another challenge is system performance.  If automation takes a back seat, then performance testing is usually behind that.  In order to deal with that challenge, having a centralized performance team will help.  Usually 2-3 performance engineers will be sufficient.  I encourage a very close look at performance testing within the Agile context.  If you don’t need it, don’t do it, if you need it, you better do it.

UI versus Web ServiceAgile software testing requires some fundamental changes in terms of how things get tested.  More and more agile software testing is taking a closer look at web service testing.  It is a critical component that can find a lot of defects.  The need for agile teams to focus on spending more time in this area is due to the amount of applications that are using API’s to transfer information.  It is a lot quicker to run hundreds of transactions through an API versus through the UI.  Agile software testing teams who are able to make this transition are simply more effective.

Let’s face it, transforming your software testing organization is going to be very challenging.  Agile software testing is extremely challenging.  The key is to have a strong team who is flexible and is adaptable to constant change.  The teams that can do that most effectively will be successful.

If you are looking for more additional information on Agile roles you can find additional information here:

Role of an Agile QA Manager

Role of an Agile Development Manager

Role of an Agile QA Engineer

Role of an Agile Business Analyst