Let’s face it, writing detailed test cases takes a lot of time and effort. As a tester, I know this is very tedious work. However, I know first hand there are some tremendous benefits that far outweigh the time involved. It certainly is not easy, but if planned out properly can be done extremely efficiently. You will probably get some push back in certain areas and using certain methodologies but it is extremely important in my opinion. Agile for example, is not in favor over detailed documentation.
Here are 7 Great Reasons to Write Detailed Test Cases
- Planning: It is important to write detailed test cases because it helps you to think through what needs to be tested. Writing detailed test cases takes planning. That planning will result in accelerating the testing timeline and identifying more defects. You need to be able to organize your testing in a way that is most optimal. Documenting all the different flows and combinations will help you identify potential areas that might otherwise be missed.
- Offshore: If you have an offshore team, you know how challenging that can be. It is really important to write everything out in detail when you communicate offshore so that everyone understands. It is critical to write detailed test cases is no different. Without those details, the offshore team will really struggle to understand what needs to be tested. Getting clarifications on a test case can often take a few days of back and forth and that is extremely time consuming and frustrating.
- Automation: If you are considering automating test cases, it is really important to have all the details documented. Automation engineers are highly technical but they might not understand all the flows of the application, especially if they have not automated that application before. Without the details, there is a high possibility that steps will get missed and perhaps that will cause the automation scripts to not be written properly.
- Performance: The performance engineers must also write performance test scripts. They also are more technical in nature, but they really struggle to get the right amount of information needed. It really helps the performance test engineers to have document test case steps so that they will be able to create their performance test scripts a lot faster.
- Audit: I have had the experience in testing applications that fall within domains which require regulatory compliance such as telecommunications and insurance. These domains require internal and external audit teams to review all testing activities. It is important to have the teams write detailed test cases so that audit will have a solid understanding of what is tested and will minimize the amount of questions that will eventually come back to the testing team.
- Development: I have found that having detailed test cases will help the development team, especially when there are defects, to provide additional guidance and direction. This helps to accelerate the fix time and thus the ability to retest and close those defects.
- Training: I have found that it is extremely helpful to have detailed test cases in order to train new testing resources. I typically will have the new employees start understanding how things work by executing the functional test cases. This will help them come up to speed a lot faster than they would be able to otherwise.
As you can see, there is valid justification to write detailed test cases. I am sure if I spend more time, I will be able to come up with another 7 great reasons. I hope this information is helpful and will encourage you to write more detailed test cases in the future.
Creating Predictive Analytics for Quality Engineering
If you are in the IT profession, you know that metrics are extremely important in helping to make decisions. This is also especially true for Quality Engineering teams. 10-15 years ago, testing was primarily conducted by software quality analysts and test cases were executed manually. Most software testing teams were small, and they would run a limited number of test cases to ensure things worked. Using this approach, it was relatively easy to know if there software was ready for production, and that QA manager could pull the team into a room and determine if the software was ready to be deployed. Those times have drastically changed.
Here are a few reasons why software testing has evolved:
- There is a lot of software testing tools that enable status reporting
- Automation and Performance testing tools are widely utilized
- Applications are more complex and tightly integrated using interfaces across multiple technologies
- There is tremendous pressure to deploy products quickly to market
- Testing applications earlier in the lifecycle (shift left)
- Distributed teams
There is a need based upon this evolution to have software testing metrics in order to make better decisions. This data needs to be consistently captured and analyzed. It is important to create predictive analytics so that you will be able to determine the current state of the quality engineering effort and accurately predict what would happen in production.
Quality is required.
Speed is required.
Resources and time is limited.
Decisions must be made.
Software must be deployed to production.
In order for these things to happen data analytics must be performed. A base set of data is needed. Some of those data elements include:
Planned/Executed test cases
Manual vs Automated tests
Root Cause Analysis
Once this data has been identified, it needs to be captured and segregated. When that information is gathered, you will be able to start and see trends. If you are testing a certain application, you will be able to predict how long it will take to perform testing, how many defects you plan to identify, and most likely how many defects will make it do production. Predictive analytics will evolve over a period of years. Many companies have started using AI/Machine Learning in helping perform this analysis.
This is also a continuous process. It is something that is not done once and completed. Additional metrics and more information will be needed. Those metrics will have to be captured and predictive analytics models will need to be created or modified.
Digital transformation requires that quality engineering teams transform how testing is planned, executed, and measured. The key to digital transformation is a focus on the customer. This requires that the quality engineering teams truly understand the business, and more importantly can accurately predict customer behavior. Issues such as usability, compatibility, performance, and security are extremely crucial. Provided these issues are tested, and the results are acceptable, this will create a really positive customer experience. For example, if a mobile application is slow, the customer is not going to have patience and will quit using it.
Predictive analytics can be used for defects. Here is some helpful information that will improve quality:
- Type of defect
- What phase was the defect identified
- What is the root cause of the issue
- What changes need to be made so that defect will not make it into production
- Is the defect reproducible?
Once this is understood, changes can be made to prevent similar issues from occurring. Using these predictive analytics, overall quality will greatly improve and speed to market will accelerate. It is important to have the right amount of data so that predictive decisions can be made.
The world of software quality has changed tremendously over the last 5 years. There are many reasons why this has happened, and it is critical that education serve as the primary strategy to influence change in an organization. Here are a few critical areas where the CIO can gain a better understanding of some of the challenges that impact software quality.
Primarily due to Agile, the robust requirements that used to be a cornerstone of the waterfall methology have been thrown in the trash. While there are some organizations that continue to document and provide best practices gathering requirements, most organizations feel this is outdated and no longer necessary. The lack of proper documentation and requirements have a direct correlation on software quality. Here are some specific reasons
- Without proper documentation a developer will code software based upon their understanding. This often will result in buggy code and requires rework after production, which will be very expensive to fix.
- Without proper documentation a tester will write test cases based upon their understanding. This often will result in test cases that have to be written again and will result in the tester missing defects that will go into production.
- Without proper documentation, the test automation engineer, will build automation test cases which will have to be changed once the manual test case changes, and will miss defects that go into production.
- Without proper documentation, the production support developer will fix problems in production and will break other production code, because they didn’t get an accurate picture from the developer that originally built the code.
These are a few examples, but you should start understanding why requirements are critical.
Agile has changed the approach on how software is delivered into production. It has some tremendous benefits, and done properly, it can greatly increase productivity within an organization. It is quick, lean, and provides fantastic feedback from the business. CIO’s love it because it provides rapid return on investment.
There are some challenges from a software quality perspective that need to be incorporated and education needs to happen across all levels of an organization. Unless you are deeply entrenched on an agile team, you will probably make a ton of assumptions that are incorrect. Within an agile team, everyone has a responsibility for software quality. Here are some areas that will have a direct impact on software quality:
- Agile Stories must be well written. It is not enough to throw out tasks without enough detail.
- Agile Planning is critical. There is some real misunderstanding about agile as it relates to planning. The more planning and organizing that can be done, the better the team will respond and be able to pack more work within a given sprint.
- Documentation is needed. This is another area which is often misunderstood. Providing documentation allows the team to understand details and more effectively code and test the desired solution.
- Developers must still test. This is important. Just because the agile team has a tester, doesn’t mean that a developer doesn’t have to test.
- All testing can be automated. Well, perhaps it could. But it might not make sense, especially if the code isn’t stable and will need to change over sprints. ROI, is still important within agile, so just because you can automate a test, doesn’t mean that you should. This is the most misunderstood item that CIO’s need software quality education.
CIO’s often ask why there are so many defects found in production. Well, that is a fairly complicated question. In order to answer that, a full analysis will need to be done on the defects to gain a better understanding. Many years ago, when I began a new job, the same question was asked. In order to come up with the correct answer, the CIO brought in an outside company to perform a software testing assessment. While I was fairly new, it wasn’t uncommon for this to occur. In fact, I welcomed the opportunity, because I already had a hypothesis as to why this was happening. Typically this is primarily due to little or very poor requirements. The company came in and did the assessment and found that there were 38% of defects that were making it into production. That is a really high percentage, most companies will average around 5%. Over a period of time, we started to tackle the problem, and after 1 year of work, we were able to reduce the production defect leakage to 5%. This was a tremendous accomplishment and required a team effort from project managers,business analysts, developers, and testers to make this happen.
Software Quality Metrics
CIO’s are very metrics driven. They use data everyday to make better decisions. While there is usually some form of metrics around software quality, it usually does not make it into the CIO’s hand for one reason or another. I believe software quality metrics will tell a story, and provide great insight to those that are willing to look and interpret the data. Several years ago, my team and I started to perform analysis on what data was important and which metrics would help us make decisions. Once we agreed on what those metrics would be, we started to gather that information release over release. We started to see trends that would help us test more thoroughly those areas which where problematic. That resulted in bringing production defects down. We also, built a web based dashboard, that would allow the CIO and anyone else in the company to see how testing was progressing. Using this dashboard, we could determine if we were going to meet our testing timelines, and see what outstanding defects were holding up production deployments. This was a true game changer for the organization.
Educating CIO’s on sofware quality will take time. CIO’s want to have high quality software, they often don’t understand how to get there. They don’t want their business partners to suffer through using software that doesn’t work properly. It is important as a quality champion, you spend time with your CIO and provide software quality education, so that you can avoid having significant issues in production. Software quality can be done effectively and efficiently within an organization.
Build Strong Edge Test Cases
If you are a software engineer, there is a lot of effort in software testing. With management demanding more and more quality, there is a strong push to create efficient test cases which prevent defects occurring in production. In my 15+ years of software testing, I have found that most organizations do a very good job in covering your happy path scenarios. I have however found that creation of negative test cases and development of edge test cases are very limited. The main reason for this is that there is often very little time but more importantly, a lack of creativity to build these scenarios.
Creating strong edge test cases requires a very creative mind. Sure, you need to be able to understand how the system works, but you also need to think outside the box and ask the hard questions in your test case workflows. If you only go by a rigid set of requirements and never deviate outside of that you aren’t going to find any edge scenarios. Building strong edge test cases requires solid application knowledge. For example, it is important to know what will happen when the same user tries to access the same information and perform an update on it. Will the record get locked? Which person will update the record? These are the types of questions that have to be answered. Here are some additional potential edge scenarios:
- Have a user login and disable the user to see what happens
- Have two users try to update the same record
- Disable the connection between the application and database
- Have the same user try to login from two different computers
There are many more possibilities when building edge scenarios. Over a period of time, you can begin to identify these edge test cases a lot easier, and you will begin to see tendencies which will cause these conditions. Chances are pretty great that if the application will allow you to do something, your business user is going to try it. These edge scenarios are also the ones that the development team is not typically going to think about, so they often will not program for it, and it will really require some thinking on their part. These edge test cases will often stir up a lot of controversy, because these are often things that are not spelled out in the requirements. Some of them will usually result in some significant frustration from a business perspective because it can cause a lot of uncertainty and could impact downstream processes because it wasn’t identified.
Edge test case can also typically be negative scenarios. They could be automated, but they may not be the best candidates, because they are often complex in nature. These are the types of things that requires deep thinking and creativity. The testers that are the most creative, will always strive to get to the edge of coverage and push beyond in order to prevent business users from finding defects. The good thing about edge testing is that you can perform these types of test in waterfall, agile or other SDLC cycles.
If you are a manual software tester, you have probably read or been informed by your boss that manual testing is slowly dying and that you need to upgrade your skills and become an automation engineer. I don’t think this can be further from the truth. While it is true that there is value in upgrading your skills and learning test automation, manual testing will always be a valuable skillset.
Why Manual Testing Isn’t Dead
- Manual testing can uncover some significant defects through exploratory or adhoc testing
- Test cases are typically built by the manual tester which requires analysis and a deep understanding of the application being tested
- If test cases are automated, they will typically first be created by the manual tester and then automated
- There are some test cases can’t be automated. Test cases that require analysis of bills for example shouldn’t be automated
- Some test cases shouldn’t be automated because it doesn’t provide a good return on investment
- A manual tester sees trends and typically knows where the defects are found
- When schedules are compressed, the manual tester brings value in prioritizing what is tested
Recently with things such as machine learning and AI tends to surface the question again. While machine learning and AI can be valuable additions, I don’t see the value of the manual tester decreasing anytime soon. Learn why manual testing isn’t dead. Many software testing organizations have some manual testing rockstars and they wouldn’t trade those valuable resources for anything.
With software testing, and agile testing in particular, it is critical not to forget about the negative test scenarios. Agile forces teams to move a break neck speeds and often we are lucky to get the happy path scenarios tested, so you are probably thinking how in the world can we also have time to cover the negative test scenarios. Unfortunately, when there are production quality issues, the root of most of those defects is that the test scenario was not tested. Planning in agile, contrary to what many may think is that test planning is a critical piece of the overall process of quality.
I have been running an agile team for over a year now, and we have done a great job at covering happy path scenarios, or those items which we identified which need to be tested. However, we have done a very poor job of identifying and documenting negative test scenarios. For our product, we manage electronic file transfers between our customers and other internal systems. With this one scenario, two different files were received at the same time, and the second file ended up overwriting the first file. While we were able to reprocess the overwritten files, it was a painful lesson because we did not factor that negative test scenario in our testing.
While it is impossible to account for all the different negative test scenarios, it is possible to document those which might be most probable and automate as much of those scenarios that you can, so that you can speed up the testing process. In the event that defects are discovered like it happened on my team, it provides a great learning opportunity to grow and raise awareness which will help identify potential missed opportunities. I would venture to guess that most agile testing covers 90 to 95 percent of happy path scenarios. If we spend more time and identify 5 to 10 percent more negative test scenarios, then I believe our overall quality will increase.