Software Testing | Agile | DevOps | Quality Engineering

Welcome to the QA Revolution!

What is the QA Revolution?

software testingSo you are probably curious about the name of this website and want to learn more about it.  I have been building this website for many years.  My passion is software testing and I really enjoy learning as software testing is changing every year.  I enjoy sharing my knowledge and passion with other people.  I am hoping this website will be helpful for you and that you will learn something when you stop for a visit.

Many years ago, while I was completing my MBA, I began to learn the importance of marketing and how it could help transform the current testing organization that I was leading.  I met with the team and discussed the idea and wanted to learn more about why we were struggling and what we needed to do to become a critical pillar within the IT organization.  We talked about some of the mistakes we had made in missing some defects, but I also encouraged them that we needed to build a brand and market it outside our team.  Everybody embraced it, and we came up with a team name and called ourselves The QA Revolution.  I wanted the team to be able to hold their heads up high and be proud of the work that they had done.  We built a quality brand and literally started QA Revolution!  We created posters, created a logo, had signature lines for our email and developed websites that would help to build our brand.  This was a tremendous hit, and because of that, suddenly being on the QA team was cool, and others wanted to come join the team.  And believe it or not, they did.  I am talking developers and engineers..the cool kids wanted to join the cooler organization.

I tell the story in order to build a point, that building a brand is important.  It helps to define who you are and where you are going.  It establishes a vision with a mission.  These days people are looking for something to believe in that they can fully embrace.  Those memories will live on forever, and I am looking forward to be able to reproduce that again and again.

Agile Transformation

The Agile methodology has turned traditional software testing organizations on its head.  We have real world experience transforming multiple industries and successfully implementing best agile testing practices.  Agile done well can have a tremendous impact.  The agile product methodology is the one that I have successfully used at one of my most recent positions.  It is important to leverage best practices when using agile.  Developing software in short agile sprints with high quality production delivery can put your organization far ahead of your competition.  There are many learning opportunities in reaching agile nirvana.  It is a process and more of a marathon than a sprint.  Some challenges I have learned along the way is there are agile mistakes along the way.  It is fine to make mistakes since no organization is perfect.  The important thing is to learn from your mistakes.  From a testing perspective, things like end to end testing are often forgotten because they take too long.  The result of that is poor quality, which is something that your business organization cannot afford.

DevOps

DevOpsDevOps has blurred the lines between development and testing.  It has accelerated the pace of automated code deployments and given opportunities to integrate automated testing into the process.  DevOps is simply the processes of collaboration between development and the operations team.  While the goal is to make processes more efficient automating everything may not be the best answer.  It has to be determined within each organization.  The ultimate goal is to help your business win.  If you have built a killer DevOps team, but don’t set your business up for success, you will have missed the goal entirely.  I believe the DevOps practices are here to stay, but I have used these same techniques for many years before the word was invented.  DevOps enables testers to spend more time testing and less time waiting for test environments to be setup.  If you are looking for more information on DevOps, I have published an article that highly recommends reading The Phoenix Project and The DevOps Handbook which are both written by the DevOps Rockstar Gene Kim.  I don’t like to read but I really enjoyed these two books.  It is critical in invest time in learning DevOps.  There are DevOps challenges that will happen as you standup the organization, so you need to be realistic and allocate enough time and resources to do it properly.  Once you reach a steady state, then you can focus on continuous process improvements which will really get you to the next level and provide some significant DevOps benefits.  DevOps is really very complementary enabling software testing transformation.

Test Automation

Test Automation has gone mainstream and all software testing organizations must do it to keep up with business demand.  It requires extreme technical expertise and multiple toolsets are needed.  Test automation is my passion and I have built multiple organizations and driven automation up to 95% automated.  I believe all software testing organizations need a strong test automation practice.  I have used both Selenium and Unified Functional Test to help deliver strong value in automating pretty much everything from mainframe to mid-range to web front ends. There needs to be a strong focus on automating what makes sense, and what will provide a good return on your investment, since it can be very expensive to maintain.  Creation of an effective test automation framework is critical so that it can handle most of the workload.  The beauty of an automation framework is that you can leverage it, instead of building thousands of scripts that will have to be individually coded.

Quality Engineering

Over the years, software testing has evolved.  It has grown from some fantastic software testers to quality engineers.  Manual testing has evolved for sure over the last 20 years into something that is more sophisticated.  Sure, there will be manual testing forever, because everything does not need to be automated, but the days of having only manual testers are long gone.  In order to keep up with the demand, you have to have highly skilled individuals with development type skills.  Notice I said development type skills, not developers, because I firmly believe that a great developer may not transform into a great tester.  The key is a mindset.  Are you curious?  Do you ask a lot of questions? Are you analytical?  Do you like numbers?  Can you see typos and other errors better than anyone else?  Will you ask the difficult questions and challenge the status quo?  These are some of the questions I ask when I talk to potential candidates.  While having a stacked resume with skills, education, and certifications are truly impressive, they will not answer the questions above.  A quality engineer needs to be able to seek, probe and find those edge cases where nobody else dares to venture.  Quality Engineering has somewhat merged with DevOps, so those lines are starting to cross.  Those that can think, code, and solve complex problems are the quality engineers who will have long careers in this field.  Are you one of those?  Are you looking to build your career into a quality engineering expert?

I have built many years of knowledge on The QA Revolution website.  I encourage you to look around and learn more.  I am constantly adding new content that is relevant to the rapidly changing technology landscape that we currently live in.

 

Most Popular Testing Tools in 2017

2017 Testing Tools

I recently conducted an informal Testing Tools survey on LinkedIn and asked the following question:

If you are a software tester, I need your help. What software testing tools does your organization currently use?

I received a significant number of replies and based upon that information I have gathered some very interesting information.  I compiled 230 responses and identified the most popular testing tools.

Big Winner: Selenium

As you can see Selenium received the largest number of votes by a wide margin.  For most people in the software testing industry, this is not a huge surprise.  With open source testing tools making significant strides in the last few years and companies looking to save expenses by going open source, it is expected that a testing tool such as Selenium would be widely used.  The big takeaway for testers: Learn Selenium.  In order to effectively use Selenium, you are going to need to learn how to learn Java programming.

Big Loser: HP ALM/UFT

If I would have asked this question 5 years ago, HP QC and QTP would have been at the top of the list.  I doubt ALM/UFT will be able to get much higher in terms of utilization, but it will be interesting to see if Micro Focus, who recently purchased these tools from HP, will be able to turn things around.  The tool stack is well integrated between ALM/UFT/LoadRunner but that isn’t enough anymore.  In my opinion the leading factor is cost.  Testing organizations simply don’t have the budgets anymore to pay large sums of money for licensing costs.  The second factor is these tools failed to keep up with the changes in technology and let smaller and leaner organizations beat them in developing better testing tools.

Web Service Testing Tools

It is interesting to see that web service testing tools such as Postman and SoapUI are slowly creeping up to the top.  I believe more and more testing organizations are moving up closer to unit testing and beginning to hit more web service testing and this is an area where testing organizations have tremendous opportunities to find defects earlier in the cycle.

Performance Testing Tools

I am not surprised by the relatively low numbers of performance testing tools currently being used by testing organizations, but I would strongly recommend companies begin to take this seriously.  When performance testing is not adequately done, it is only a matter of time before catastrophic issues occur.  Have testers forgotten about Healthcare.gov?

Other Tools

I asked a fairly broad question on LinkedIn.  It was by design.  I primarily wanted to understand what software testing tools were being used, but I also wanted to get a better idea of what tools testers use outside of testing tools.  If you take a look at the chart above, JIRA and Jenkins were heavily used tools.  Long gone are the days were testers only have to worry about learning testing tools.

I hope this information has been helpful.  I have been able to gain a better appreciation for the challenges software quality engineers face everyday.  I encourage you to take this information and learn a new software testing tool today!

 

Performance Engineering: More Art Than Science

performance engineeringIf you are responsible for performance engineering, you know there is a true art to understanding how things work.  Performance Engineering requires extensive understanding of the applications that are tested and the performance aspects of how the applications behave.  Early in my career I thought performance was based upon pass and fail criteria only but over the years I have understood that is not always the case.  Here are some performance engineering principles you must consider:

 

  1. There are many components which drive application performance including (but not limited to) CPU, Memory, Cache, Databases, Servers, Network Traffic, and Transactions
  2. It is always important to run a baseline test so you have a point of reference in determining what is acceptable
  3. I always recommend getting at least 2 acceptable performance runs in in order to eliminate any anomalies
  4. The production environment will likely be the only environment that is sized adequately, many organizations will have a dedicated performance testing environment but you will need to extrapolate your information to determine what will be acceptable in production
  5. Focus on building performance scripts that cover the top 80-90% of the most used transactions and based your performance tests on those only
  6. It would be nice to have a tool such as App Dynamics which will help in troubleshooting performance related issues
  7. Run your performance test for at least 3 hours in order to identify any performance bottlenecks
  8. Always involve development and infrastructure teams in helping with the performance analysis, they should be major participants in the review and sign off of the performance results
  9. Performance Engineers should understand how the applications work and be able to help identify performance issues
  10. It is important to view the performance test as a whole versus looking at each transaction.  In general if there are a few high transactions, that will be fine, but the overall test should be satisfactory
  11. You need to build a large end to end suite so you can see how the application or applications behave together

The list should help you gain a better understanding of performance engineering.  It is important to constantly work on gaining a greater understanding of performance engineering.  As technology changes and evolves, if will be necessary to keep up with new trends.  In addition, there are some great performance engineering tools that can help complement those tools that you already use.

Quality Engineering KPI’s

There is growing attention these days on Quality Engineering KPI’s.  Key performance indicator is defined as “A measure of  performance, commonly used to help organization and define success what is typically in terms of making progress towards its long term organizational goals.”

Key performance indicator so companies try positive results towards what an organization views as important. Many people often get confused between what a key performance indicator is versus a business metric. A business metric is defined as a “quantifiable measure businesses use to track, monitor, and assess the success or failure various business processes.” Can key performance indicator’s and business metrics be measured the same?  Absolutely. The differences the organization may not identify them as a critical measurement of their long-term goals. It is very important for organizations to have a long term goals because it helps everyone move towards ensuring these goals are met and allows the company to grow and expand. Without goals employees will struggle to understand what is important to the company. Too often companies gather tons of information  and metrics but they waste valuable time because they are unable to tie the metric to long term goals.  In addition to organizations, departments such as quality engineering need to have KPI’s that are tracked. Some Quality Engineering KPI’s examples include:

  • % automated test cases
  • Defect leakage %
  • No critical or high defects
  • 100% requirements coverage
  • 100% test case execution

It isn’t for engineering to go both key performance indicator’s and business metrics. This helps managers and testers make better decisions for the organization. Also there is a need to use the key performance indicators and metrics to ensure they are relevant to long-term goals and organizational goals. There is a balance that needs to be closely monitored to ensure too many performance indicators and metrics are not being measured. It will also help the process together these matters can be leveraged using a real-time dashboard. I hope this is helped provide a high-level overview of Quality Engineering KPI’s.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

QA challenges in DevOps

Introduction

Quality Assurance has always been an evolving discipline in software development. With the emerging trends in IT industry, the need to better understand, manage, and adopt the QA activities is increasing. With the onset of agile and lately DevOps, the way organizations develop software has changed, and so have the ways to enforce QA. Software development cycles have become short and quick. With this QA teams face new challenges as they work to keep pace. The advantages of overcoming the challenges include quality, optimization, process improvement and higher productivity.

 

Understanding QA in DevOps Landscape

DevOps advocates good principles and practices that help improve communication and collaboration between the organizational silos. This also implies to QA organization and their development counterparts. But in a DevOps scenario, the walls will be eliminated and this helps facilitate knowledge sharing, experience and specialized skills to deliver quality systems. In the era of DevOps the focus of QA teams will be more on preventing defects than finding them.

 

Challenges faced by QA teams

QA culture – In the context of DevOps, quality requires a change in how it is being conducted. This also implies an intense transference in the organizational culture as well. It is very important and also challenging to think of innovative ways of identifying unique techniques to test the software quickly and efficiently. This will enable to continuously ensure quality while also growing and evolving the QA services provided.

Facilitation of quality – From a DevOps perspective, QA team needs to understand the business for the system being verified. For this to happen, QA team should partner with business experts, including the product owner, to understand how the system being tested needs to function in order to support the business. QA teams will be disabled if not involved in those initial discussions. This involvement helps QA to become the facilitator of quality.

Collaboration – QA is the binding entity between development and operations. So QA team should be involved right from the early stages of development. This will enable them to collaborate to have software developed and supported more effectively. Also QA should be considered as responsibility of entire project team rather than the responsibility of dedicated QA team.

Early testing – One of the main objectives of testing in DevOps is early detection of defects in development cycle. For this to happen testing must begin very early in the cycle. QA should begin testing with whatever code is available even if the features are not complete. This requires lot of maturity in documenting self-sufficient user stories that do not depend on another for testing.

Test coverage – In DevOps there is a rush to deliver software quickly with the techniques like continuous integration and deployment. Also because of rapidly changing requirements, there is a possibility to miss testing critical functions.  To overcome this challenge, a thorough and detailed traceability of requirements to test functions should be maintained.

Build verification – As DevOps encourages frequent builds, there is a higher possibility of code breaking existing features.  For this reason it is not practical to have testers do these verifications manually. It is recommended to rely on automated testing for these smoke tests.

 

Conclusion

If the above discussed challenges are addressed then QA in DevOps can play a critical role in accelerating development and release schedules. DevOps guiding principles like test first, free communication and seamless collaboration help resolve some of the QA challenges and also enable the QA team to take their deliverables to the next level. In DevOps testing is a continuous process and supports the process of incorporating continuous feedback to enable better quality.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Measuring DevOps Success

Introduction

New age IT world is making revolutionary advances in ways nobody predicted. Developing applications such as the cloud, advanced analytics and expanding mobility have transformed the vision of software industry. In today’s IT world it is all about agility with quality. Traditional processes and methods of developing software are challenged with keeping up with the complexities that come with these new environments. As a result, DevOps has come to the fore as a new IT philosophy which will help overcome these complexities. The aim of DevOps is to bring collaboration among the different IT teams. DevOps is successful in increasing the agility and in faster software/application deployment. But there is no success without measurement. So DevOps success also needs to be measured.

Measurements

Measuring the success of DevOps is very important because DevOps initiatives target the very processes that determine how IT works. A metrics-oriented mindset is key to ensure that DevOps initiatives deliver the intended results. Data-driven decisions and focused improvement activities lead to increased quality and efficiency. Also the use of feedback to accelerate delivery makes DevOps a successful IT culture. With DevOps, as with any IT initiative, knowing what to measure is always the first step.

The following suggestions help identify the key measures.

Measuring Culture:

This is a tough area to create the mechanism to measure. Because it is very difficult to assign a dollar value. DevOps advocates eliminating stress, creating an environment of collaboration. This enable the people to work more effectively. Some key representative metrics are:

  • Staff retention
  • Change acceptance
  • Resource cross skilling
  • Knowledge sharing
  • Resource utilization

Measuring Agility:

Agility again is not clearly defined. One should define the measurable attributes of agility. Some of the reasons why the DevOps is fast gaining acceptance are the ability to deliver software faster, with fewer defects and faster resolution of problems. Some key representative metrics are:

  • Speed of deployments
  • Time it takes to fix failed releases
  • Frequency of releases
  • Change lead time
  • Sprint velocity

Measuring Quality:

The success of DevOps is directly related to the value and the quality delivered to the end users. The key objective of any DevOps practice should be to contribute creatively and improve the existing practices. While measuring quality using metrics one can evaluate, modify, and improve processes over time. Some key representative metrics are:

  • Production outages
  • Defect rate in production
  • Mean time to recovery
  • Deployment roll backs
  • Success rate of deployments

 Conclusion

As Lord Kelvin (Scottish mathematician and physicist) said, “Without measurement there can be no measurable improvement”. DevOps is evolved from a rich history of process improvements. Feedback and measurement are the factors that drives the improvements in DevOps. The success of DevOps is to increase business value by making it agile through continuous delivery of services that satisfy customer needs. There are several tangible and intangible benefits of DevOps. They require measurement to help an organization better appreciate the impact of a DevOps approach. If this goal is not met then whatever is being done is not working or is not right.