by admin | Mar 21, 2019 | Agile, Automation, DevOps
It is extremely important to develop a strategic plan when creating test automation scripts. With the push towards Agile and DevOps the mindset has become to automate as much testing as possible. While that idea makes sense, it is important to determine how much test automation is needed. Part of the problem is that those who are managing Agile or DevOps teams have a really different mindset, and they really don’t understand basic testing principles. Instead of taking the time to understand, they force their understanding and push test engineering in the wrong direction.
Test Automation Planning
Test automation is certainly critical for Agile and DevOps. There are things that can and should be automated but before that is done, a few questions must be asked:
- How many times are these test automation scripts going to be executed?
- Is the code stable or is it going to change a lot over the next several sprints?
- Has the test been executed manually and has it passed successfully?
- How much time will it take to maintain the test scripts?
- Do you have the right resources to build and update the test scripts?
- What type of tools are you going to use?
- What type of framework are you going to build?
- How much does the test automation software cost?
These questions will help lay the foundation of your test automation planning. The next step will be to identify the number of test cases that need to be automated.
Testing That Should Not Be Automated
Here is a list of test cases that should not be automated:
- Test cases that will only be executed a few times
- Test cases that require human touch, for example review of a customer bill
- Test cases that require a third party billing system such as a payment gateway that doesn’t have a test environment
- Test cases were the code is unstable and is constantly changing
Test Case Quality
Test automation scripts help with providing test coverage across applications. This allows faster deployments because it doesn’t require manual testing. However, if you create hundreds of test cases and those never identify any defects, is it really worth maintaining and running those over and over again? Probably not. The test cases that are automated should be able to capture defects. Therefore, it is important that you have some really sharp manual testers who can think through the testing scenarios and identify problem areas that need attention. Without that system knowledge, you will not capture defects and it will not be a good use of your resources.
Return on Investment
It is really important to understand how much cost is involved in creating automated tests. If you create thousands of them, how much is it going to take to make changes on those when the GUI or back end systems change? If it is your department or team, as a manager or employee, you really need to be able to answer this challenging question. At every organization, I make sure that I calculate the ROI. There are a few ways to do this. First, you roughly need to know what is the cost of a member of your team or the average rate per contractor or employee. Most organizations will have that already. Second, you will need to determine how many scripts are going to be automated and how many times are they going to be executed. Third, you need to know how much time does it take to execute those test cases manually. Fourth, once the test cases are automated, you need to know how much time does each one take to execute. These items will at least give you a basic ROI. Most ROI numbers are somewhere between 6-12 months. Anything over a year, probably isn’t worth it. There are other variations that are more complex, but this will at least get you started. Knowing this information is certainly powerful, and it will help justify the expense for having automation engineers.
I cannot stress the importance of doing these basic steps. With Agile and DevOps, everything tends to be rushed. Perform this basic analysis first and spend a little time up front to yield large returns in the long run.
by admin | Feb 21, 2019 | Agile, Blog, Manual Testing
If you are a manual software tester, you have probably read or been informed by your boss that manual testing is slowly dying and that you need to upgrade your skills and become an automation engineer. I don’t think this can be further from the truth. While it is true that there is value in upgrading your skills and learning test automation, manual testing will always be a valuable skillset.
Why Manual Testing Isn’t Dead
- Manual testing can uncover some significant defects through exploratory or adhoc testing
- Test cases are typically built by the manual tester which requires analysis and a deep understanding of the application being tested
- If test cases are automated, they will typically first be created by the manual tester and then automated
- There are some test cases can’t be automated. Test cases that require analysis of bills for example shouldn’t be automated
- Some test cases shouldn’t be automated because it doesn’t provide a good return on investment
- A manual tester sees trends and typically knows where the defects are found
- When schedules are compressed, the manual tester brings value in prioritizing what is tested
Recently with things such as machine learning and AI tends to surface the question again. While machine learning and AI can be valuable additions, I don’t see the value of the manual tester decreasing anytime soon. Learn why manual testing isn’t dead. Many software testing organizations have some manual testing rockstars and they wouldn’t trade those valuable resources for anything.
by admin | Jan 10, 2019 | Agile, DevOps
If you are looking to learn more information about DevOps it will take you a while to figure out what are the top 2 DevOps books on the internet today. I am not a huge fan of reading, so when I read, I really like to enjoy the books I read. In addition, you want to make sure that you are getting the right information. There are so many thoughts around DevOps, and there is quite honestly a lot of misinterpretation on the subject, whether it is at a conference, on a blog, or written in books. You want to make sure you can understand DevOps best practices so that you can effectively implement those either as a consultant or an employee for the organization you are working for. As you probably already know DevOps is one of the key cornerstones of organizations that are looking to transform their development and operations processes. Anyone you talk with today is going to mention DevOps over and over again as a part of their IT transformation strategy. You want to make sure you understand and can actively contribute to the organizational objectives. Simply put, DevOps is DevOps is a software development methodology that combines software development with information technology operations. The goal of DevOps is to shorten the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives.
Top 2 DevOps Books
Several years ago, I was at a conference, and I heard about a particular book that sounded really interesting. It isn’t your typically technology book because it is written as a fictional novel. Is is called The Phoenix Project. It really is a fantastic book and quick, easy read. The book helps you to identify with key challenges and issues that are facing most corporations from both a business and a technology perspective. Here is a high level summary of the book.
Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.
With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.
In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.
It really helps to be able to understand the challenges that Bill faces, and the steps he and others take in order to overcome the daily struggles. The organization is constantly putting out fires, so they are unable to get ahead of these issues and develop a long term strategy. I have personally been in several organizations, and part of the reason why I really enjoyed the book was that I could identify with the challenges that the characters in the book were facing. The Phoenix Project authors (Gene Kim, Kevin Behr, George Spafford) got together and wrote the book leveraging the ideas and concepts from a prior book call The Goal, which I also recommend.
The second book which I highly recommend is The DevOps Handbook. This book is more of what you would typically expect from a technology book. It helps you to gain a much better understanding of what DevOps is, and what steps you can take to implement DevOps in your organization. The authors (Gene Kim, Jez Humble, Patrick Debois, John Willis) are all industry heavyweights and really have perfected the art of DevOps and are helping others to do so. Here is a high level summary of the book.
More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.
And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.
Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.
I think a combination of both of these books will help you gain a tremendous understanding of DevOps. After the initial reads, I keep both of these books handy, and I have read them at least a few times. In addition to that, I have recommended them multiple times to everyone who is interested in learning more about this software development practice.
by admin | Sep 5, 2018 | Agile, Blog
I am an avid agile fan and hands on agile program manager. I started my journey almost two years ago, and my team and I have learned some valuable lessons along the way. Currently we have doubled our velocity and increased our skills as a well oiled agile machine. Have we have issues? You bet. I am hoping you can learn some valuable lessons from my experience. Here is a list of Top 5 Agile Pitfalls:
Top 5 Agile Pitfall #1: Lack of Integrated Software Testing
Lack of integrated software testing is especially dangerous when companies have multiple products and are running multiple agile teams. The agile teams are often heads down and focused on their product only, and don’t have the time, energy or effort to understand how the systems interact with each other and what potential pitfalls are downstream. Most systems have dependencies with each other in terms of data or interfaces. Product teams today have subject matter development and testing experience, but may not understand other systems and how those interactions. If there are understood impacts, the responsibility usually is handed over to the team who is responsible for that product. It usually occurs with little to no planning, and typically little warning is done in advance. Something to the effect of: “hey can you perform some regression test this feature real quick and make sure it works, and provide me a sign-off before the end of the day since we are releasing into production tomorrow?” If you are in testing and a part of an agile team you understand that agile has a lack of integrated software testing.
Agile software development works great for the most part, but there are some pitfalls that will happen. Agile works well with a single product because there is essentially no integration points, however, most enterprise systems are going to have integrations with either other internal applications or third party applications. It is not sufficient enough to assume that is it is the responsibility of another product or team to ensure the quality is going to work from an end to end perspective. You certainly don’t want to have your business partners or end customers find out that something does not work properly.
I currently manage an agile team and my direction to them is to get the necessary access to all the interfacing systems and run the test to make sure it works properly. Our application is the upstream system, so if something goes wrong, we will be the group that will be held accountable, regardless if it is our system. Sure, it takes additional time and effort to perform that testing, but as long as we account for the work in the sprint, we are covered. I would much prefer to test it ourselves and be sure rather than suffer the consequences.
I encourage you to eliminate agile pitfall #1: agile has a lack of integrated software testing. If you can avoid this pitfall, you and your agile team will be successful!
Top 5 Agile Pitfall #2: Too Much Technical Debt
Using the agile methodology, software can be developed very quickly, in fact, business needs demand it is produced as quick as possible. While there is nothing inherently wrong with that, it is important to develop it efficiently. Agile teams want to build the best designed and highest quality product possible. However, that isn’t always what happens. My team has recently been involved in leveraging a COTS product called Globalscape to consolidate all our B2B file transfer systems into a single solution. In fact, our company was acquired, so between both legacy companies we had four solutions which were being used. We selected Globalscape, but the product was brand new to our team. We didn’t have extensive experience but were driven to move off the other systems. We made the best decisions that we knew at the time, but looking back, there were some things that if we had better information and experience with the product, we would have certainly done differently. Naturally, we accumulated technical debt, and we created stories in our backlog to handle those. In certain sprints, we had to put more technical stories in place to implement the necessary changes. While our business partners weren’t thrilled, they understood those needed to be done.
In some situations, you can address technical debt issues as you are addressing stories in the backlog. For example, with our file transfer product, we created a generic function to address how email notifications were delivered, which was much more efficient. While we were moving other B2B files into the new platform, we would go ahead and make those changes, so that we would not have to revert and make those technical debt changes later.
It is also a good idea to keep 5 to 10 percent of your sprint capacity available to handle technical debt issues. If you have one story per sprint handling technical debt, then it will make it a lot easier in not having to deal with too much technical debt later.
Hopefully this real world example will help you avoid Agile Pitfall #2: Too Much Technical Debt!
Top 5 Agile Pitfall #3: Agile Team Silos
Let’s face it, in most enterprise level software organizations, if you are practicing agile, the agile team that is developing a specific product is going to be focused on one thing: the outcome of the product they are developing. While there is nothing inherently wrong with that, there is a dangerous pitfall that can affect the whole organization. The agile team can become a single silo. I have seen it first hand, and I know it happens more often that agile experts would like to admit it. Each agile team has pride in the product they are developing. They want to be the best team that produces the highest software product. Sometimes the team can put on blinders and be so focused on meeting their sprint milestones, that they will completely forget about how their product integrates with other products and what the downstream impacts are. Some teams can even become selfish and demanding toward other product teams and expect teams to do things which are not inherently their responsibility. Agile software development at the enterprise level often creates these silos. Contests between teams such as awards given out for the best team, can arbitrarily create silos, and thus driving perception that agile teams can be self sufficient and they don’t require collaboration between teams. This is really dangerous. It is important to have an enterprise level view, and keep the big picture in mind. Agile teams work for one company, so that always needs to be the focal point so that the company will be successful. The companies that avoid pitfall #3: agile team silos and keep the company in mind, will be the most successful.
Top 5 Agile Pitfall #4: Too Focused on Agile Team Roles
Agile teams do a really great job of focusing on delivery. Each agile team typically has a Scrum Master, BA, Developers, and QA resources. There could be some variances to those depending on how your organization uses agile, but those are the most common. Each role typically has tasks and duties that are consistently carried out during the sprint. Here are how we currently use the agile team roles:
Scrum Master: The role is responsible for helping to keep things moving and helps to overcome obstacles
BA: The glue of the team. This role helps to keep things defined and assists providing knowledge to the team
Developer: This role helps to build the product and provide technical design and solutions
QA: This role helps to ensure the product is working properly and works to identify product defects
Inherently in the agile process, these roles help to get things done on the team. Sometimes, this can cause issues and the scrum master will need to step in and remove blockers. Most agile teams are fairly small, usually from 6-8 team members. During a sprint there might be some resources who need to take an unplanned absence or vacation. This often will require that other team members step in and perform other roles. My team does a great job in jumping into other roles when the person who normally performs the role is either busy or not available. You will typically see a BA jumping in to help with testing, a QA resource jumping in to help with BA work, and a Developer jumping in to either help the BA or QA resource. This helps to keep things running smoothly and keep the agile goals on track for a given sprint. I have also seen the other way, where a QA resource may take off during the first and second week of the sprint, and the team waits until the person returns to get everything completed in the last week. That is simply asking for trouble and it usually results in stories not getting completed on time.
Avoiding Agile Pitfall #4: Too Focused on Agile Team Roles will help set your agile team up for success!
Top 5 Agile Pitfall #5: Folks, It Isn’t All About Velocity!
Perhaps one of the most valuable lessons for me personally was to understand it isn’t all about agile velocity. There are other items that are more important, with one of those being quality. Now, my agile team has successfully doubled their sprint velocity over the past year, which is a tremendous accomplishment. We have some very aggressive targets because we were consolidating systems into a single platform and those were date dependent based upon some licensing renewals.
Our organization was acquired by a smaller company and they were determined to put an agile product model in place. I viewed this as a tremendous opportunity, and I was responsible for managing one of those teams. There were certain measurements the agile teams were rated on and one of those was agile velocity. Agile velocity is important because it helps you be able to predict how much work you can consistently implement within a given sprint. Our initial targets weren’t too aggressive, so we were able to consistent deliver on what was planned sprint over sprint. The agile team began to experiment and I really challenged the team to go above and beyond and it was too much for the team to handle at that point, so we ended up dialing back to a more realistic target.
While Agile Velocity was an important metric, we failed to ensure that one of our stories was working in production before we started the next sprint. This resulted in a pretty major quality issue which we eventually recovered from, but the agile team’s reputation was tarnished at that point. It took several months for us to regain our confidence and move past this issue. My most humble recommendation is that your team focuses on all the different elements, especially quality so that you won’t have issues down the road so that you can avoid: Agile Pitfall #5: Folks, It Isn’t All About Velocity!
I hope this information has been helpful and I believe you can gain some valuable knowledge about how you can avoid my Top 5 Agile Pitfalls.
by admin | Sep 5, 2018 | Agile, Blog
Agile is great, and I am an avid fan. My team has been successfully using it for a few years now and we have been able to successfully make the transition. There are some things that I have learned about agile and some pitfalls that your agile team can avoid by learning from my mistakes.
Agile Pitfall #1: Lack of Integrated Software Testing
Agile Pitfall #2: Too Much Technical Debt
Agile Pitfall #3: Agile Team Silos
Agile Pitfall #4: Too Focused on Agile Team Roles
Agile Pitfall #5: Folks, It Isn’t All About Velocity!
Perhaps one of the most valuable lessons for me personally was to understand it isn’t all about agile velocity. There are other items that are more important, with one of those being quality. Now, my agile team has successfully doubled their sprint velocity over the past year, which is a tremendous accomplishment. We have some very aggressive targets because we were consolidating systems into a single platform and those were date dependent based upon some licensing renewals.
Our organization was acquired by a smaller company and they were determined to put an agile product model in place. I viewed this as a tremendous opportunity, and I was responsible for managing one of those teams. There were certain measurements the agile teams were rated on and one of those was agile velocity. Agile velocity is important because it helps you be able to predict how much work you can consistently implement within a given sprint. Our initial targets weren’t too aggressive, so we were able to consistent deliver on what was planned sprint over sprint. The agile team began to experiment and I really challenged the team to go above and beyond and it was too much for the team to handle at that point, so we ended up dialing back to a more realistic target.
While Agile Velocity was an important metric, we failed to ensure that one of our stories was working in production before we started the next sprint. This resulted in a pretty major quality issue which we eventually recovered from, but the agile team’s reputation was tarnished at that point. It took several months for us to regain our confidence and move past this issue. My most humble recommendation is that your team focuses on all the different elements, especially quality so that you won’t have issues down the road so that you can avoid: Agile Pitfall #5: Folks, It Isn’t All About Velocity!
by admin | Sep 5, 2018 | Agile, Blog
Agile has some great benefits and I am an avid fan. I have been using agile for several years now, and I have also learned there are some pitfalls which you should avoid. Here are some of the ones that I have experienced:
Agile Pitfall #1: Lack of Integrated Software Testing
Agile Pitfall #2: Too Much Technical Debt
Agile Pitfall #3: Agile Team Silos
Agile Pitfall #4: Too Focused on Agile Team Roles
Agile Pitfall #5: Folks, It Isn’t All About Velocity!
Agile teams do a really great job of focusing on delivery. Each agile team typically has a Scrum Master, BA, Developers, and QA resources. There could be some variances to those depending on how your organization uses agile, but those are the most common. Each role typically has tasks and duties that are consistently carried out during the sprint. Here are how we currently use the agile team roles:
Scrum Master: The role is responsible for helping to keep things moving and helps to overcome obstacles
BA: The glue of the team. This role helps to keep things defined and assists providing knowledge to the team
Developer: This role helps to build the product and provide technical design and solutions
QA: This role helps to ensure the product is working properly and works to identify product defects
Inherently in the agile process, these roles help to get things done on the team. Sometimes, this can cause issues and the scrum master will need to step in and remove blockers. Most agile teams are fairly small, usually from 6-8 team members. During a sprint there might be some resources who need to take an unplanned absence or vacation. This often will require that other team members step in and perform other roles. My team does a great job in jumping into other roles when the person who normally performs the role is either busy or not available. You will typically see a BA jumping in to help with testing, a QA resource jumping in to help with BA work, and a Developer jumping in to either help the BA or QA resource. This helps to keep things running smoothly and keep the agile goals on track for a given sprint. I have also seen the other way, where a QA resource may take off during the first and second week of the sprint, and the team waits until the person returns to get everything completed in the last week. That is simply asking for trouble and it usually results in stories not getting completed on time.
Avoiding Agile Pitfall #4: Too Focused on Agile Team Roles will help set your agile team up for success!
