One Key Metric Should Drive Quality Engineering

quality engineering metrics If you are in an IT organization, you know how important Quality Engineering metrics are.  Gone are the days that you can talk to a few quality engineers and get their gut feel on determining if a software application has a high degree of quality.  It requires a lot more effort and energy and numbers to figure that out.  Quality Engineering metrics are the heartbeat of any IT organization.  While you should have several there is one that you should spend the majority of your time and effort focusing on.  That Quality Engineering metric is: Defects.  Defects tell so much of the story.  Once you are able to gather that metric and classify it you can do some pretty amazing things.

I have had many quality engineering positions over the years and understanding defects is the first one that I put my energy and effort doing research.  I start to ask a few questions:

  1. How does the organization feel about defects?  Is it seen as a positive tool or a negative one?  Do developers take defects personally or do they encourage their quality engineering counterparts to create defects?  This is a really important piece of information because it will help me to understand a lot about an organization and their appetite to influence change.
  2. Are all defects entered into a central tool?  This is necessary so that you will be able to capture all defects and not have to hunt through multiple applications to find them.
  3. How much technical debt does an organization have?  From what I have researched, most organizations carry a good bit of technical debt.  They are reluctant to spend time and energy in resolving defects.  It creates a negative experience from a business perspective and internal customers often have to workaround issues to get their desired result.
  4. Is there a standard for defects?  Once defects are being captured, there are certain criteria that needs to be gathered on each defect so that you can start to see trends and make decisions.  Some of those standards include severity, business priority, root cause, project or sprint, environment, and application.  By gathering this information you can start to classify defects based upon that criteria.
  5. Are defects being captured in production?  This is critical.  This metric will help you understand if the applications are stable, and if defects are being captured prior to a production deployment.  Often, production defects are captured in a separate tool, which makes it very hard to consolidate and gain access to for the quality engineering organization.  If they are being captured, what information is gathered?  Is it possible to tie it to a specific release or feature?
  6. Which teams are finding the majority of the defects?  Once I can get my hands on this information, I find it extremely helpful.  In one of my previous companies, I did analysis and found that most of the defects were being captured by UAT testers.  This led me to infer that they had the most subject matter expertise on the applications that were being tested.  I began to build a relationship with that team, and did several things to help the UAT testers and gain additional knowledge from that team.  The first thing, was to review the test cases they had created.  While they were at a very high level, my QA team was able to gain some valuable information.  We took that information and incorporated it into our test cases.  Second, we looked at their test cases and mapped those to our test cases.  My team had started automating test cases, so let let the UAT testers see the execution of those scripts and they agreed to let us run the regression test cases for them.  This was a huge boost in productivity for them and it really helped to solidify the relationship.

Using this framework, I did analysis on a company where I previously worked and identified a defect leakage percentage of 38%.  This number was mind blowing and really unacceptable. I established a goal to reduce defect leakage in production and set the target at 8%.

Here were some key focus areas when the team spent the bulk of their energy:

  • Reviewed all production defects and developed test cases for those.  Once the test cases were built we incorporated those test cases into our regression suite.
  • Leveraged the information gained from the UAT testers to build robust Quality Engineering test cases
  • Continued to build more test automation scripts so we could spend more energy building and executing test cases
  • Partnered with the development organization and ran our automated scripts sooner in the lifecycle so that we could find more defects upfront

After a year of hard work, the results were impressive.  We were able to get the production defect leakage down to 7%.  This was a huge milestone and everyone was thrilled.  The business was really happy with the improvements and became a fan of the quality engineering team.  While there are many quality engineering metrics that should be captured, defects is the first one that you should start with.

ISTQB Agile Tester Certification

ISTQB agile testing certification ISTQB Agile Tester Certification

If you are an interested in obtaining more information about the ISTQB Agile Tester Certification, you have come to the right place.  Most projects today have moved from Waterfall to Agile, so it is important that you have the right information to leverage best practices when testing on an Agile project.  Once you understand Agile concepts and how testing should really be done, you can provide some tremendous education to your peers and other agile team members.

The certification for ISTQB Agile Tester Certification is designed for professionals who are working within Agile.  It is also for professionals who are planning to start implementing Agile methods in the near future, or are working within companies that plan to do so, The certification provides an advantage for those who would like to know the required Agile activities, roles, methods, and methodologies specific to their role.

The ISTQB Agile Tester Certification qualification is aimed at four main groups of professionals:

1. Professionals who have achieved in-depth testing experience in traditional methods and would like to get an Agile Tester Certificate.

2. Junior professional testers who are just starting in the testing profession, have received the Foundation Level certificate, and would like to know more about the tester’s role in an Agile environment.

3. Professionals who are relatively new to testing and are required to implement test approaches, methods and techniques in their day to day job in Agile projects.

4. Professionals who are experienced in their role (including unit testing) and need more understanding and knowledge about how to perform and manage testing on all levels in Agile projects.

These professionals include people who are in roles such as testers, test analysts, test engineers, test consultants, test managers, user acceptance testers, and software developers. This ISTQB Agile Tester Certification may also be appropriate for anyone who wants a deeper understanding of software testing in the Agile world, such as project managers, quality managers, software development managers, business analysts, IT directors, and management consultants

Prerequisite:  You must have the ISTQB CTFL Foundation Level certification

Exam: 1 hour with 40 multiple choice questions

Pass Rate: 65%

Exam Registration:  Click here to register for the ISTQB Agile Tester Certification exam.

Exam Cost: $199 USD

Recommended Book:  Agile Testing Foundations: An ISTQB Foundation Level Agile Tester guide

Syllabus:  In order to pass the exam, you must study the syllabus and understand the material.  Click here to download the syllabus.

Sample Exam:  It is always a great idea to review the sample exams so that you can get familiar with the types of questions that you will see on the test.  The more questions you can review, the more confident and prepared you will be for your exam.  Click here for sample questions and click here for sample answers.

Outline:  Here is a basic outline of the material you must know in order to successfully pass the ISTQB Agile Tester Certification exam.

Chapter 1: Agile Software Development

 The tester should remember the basic concept of Agile software development based on the Agile Manifesto.

 The tester should understand the advantages of the whole-team approach and the benefits of early and frequent feedback.

 The tester should recall Agile software development approaches.

 The tester should be able to write testable user stories in collaboration with developers and business representatives.

 The tester should understand how retrospectives can be used as a mechanism for process improvement in Agile projects.

 The tester should understand the use and purpose of continuous integration.

 The tester should know the differences between iteration and release planning, and how a tester adds value in each of these activities.

Chapter 2: Fundamental Agile Testing Principles, Practices, and Processes

 The tester should be able to describe the differences between testing activities in Agile projects and non-Agile projects.

 The tester should be able to describe how development and testing activities are integrated in Agile projects.

 The tester should be able to describe the role of independent testing in Agile projects.

 The tester should be able to describe the tools and techniques used to communicate the status of testing in an Agile project, including test progress and product quality.

 The tester should be able to describe the process of evolving tests across multiple iterations and explain why test automation is important to manage regression risk in Agile projects.

 The tester should understand the skills (people, domain, and testing) of a tester in an Agile team.

 The tester should be able to understand the role of a tester within an Agile team.

Chapter 3: Agile Testing Methods, Techniques, and Tools

 The tester should be able to recall the concepts of test-driven development, acceptance testdriven development, and behavior-driven development.

 The tester should be able to recall the concepts of the test pyramid.

 The tester should be able to summarize the testing quadrants and their relationships with testing levels and testing types.

 For a given Agile project, the tester should be able to work as a tester in a Scrum team.

 The tester should be able to assess quality risks within an Agile project.

 The tester should be able to estimate testing effort based on iteration content and quality risks.

 The tester should be able to interpret relevant information to support testing activities.

 The tester should be able to explain to business stakeholders how to define testable acceptance criteria.

 Given a user story, the tester should be able to write acceptance test-driven development test cases.

 For both functional and non-functional behavior, the tester should be able to write test cases using black box test design techniques based on given user stories.

 The tester should be able to perform exploratory testing to support the testing of an Agile project.

 The tester should be able to recall different tools available to testers according to their purpose and to activities in Agile projects.

I hope this information has been helpful.  I wish you the best of luck as you prepare and pass your ISTQB Agile Tester Certification!

 

 

 

Educating CIO’s on Software Quality is Critical

software testingThe world of software quality has changed tremendously over the last 5 years.  There are many reasons why this has happened, and it is critical that education serve as the primary strategy to influence change in an organization.  Here are a few critical areas where the CIO can gain a better understanding of some of the challenges that impact software quality.

 

Requirements

Primarily due to Agile, the robust requirements that used to be a cornerstone of the waterfall methology have been thrown in the trash.  While there are some organizations that continue to document and provide best practices gathering requirements, most organizations feel this is outdated and no longer necessary.  The lack of proper documentation and requirements have a direct correlation on software quality.  Here are some specific reasons

  • Without proper documentation a developer will code software based upon their understanding.  This often will result in buggy code and requires rework after production, which will be very expensive to fix.
  • Without proper documentation a tester will write test cases based upon their understanding.  This often will result in test cases that have to be written again and will result in the tester missing defects that will go into production.
  • Without proper documentation, the test automation engineer, will build automation test cases which will have to be changed once the manual test case changes, and will miss defects that go into production.
  • Without proper documentation, the production support developer will fix problems in production and will break other production code, because they didn’t get an accurate picture from the developer that originally built the code.

These are a few examples, but you should start understanding why requirements are critical.

Agile

Agile has changed the approach on how software is delivered into production.  It has some tremendous benefits, and done properly, it can greatly increase productivity within an organization.  It is quick, lean, and provides fantastic feedback from the business.  CIO’s love it because it provides rapid return on investment.

There are some challenges from a software quality perspective that need to be incorporated and education needs to happen across all levels of an organization.  Unless you are deeply entrenched on an agile team, you will probably make a ton of assumptions that are incorrect.  Within an agile team, everyone has a responsibility for software quality.  Here are some areas that will have a direct impact on software quality:

  • Agile Stories must be well written.  It is not enough to throw out tasks without enough detail.
  • Agile Planning is critical.  There is some real misunderstanding about agile as it relates to planning.  The more planning and organizing that can be done, the better the team will respond and be able to pack more work within a given sprint.
  • Documentation is needed.  This is another area which is often misunderstood.  Providing documentation allows the team to understand details and more effectively code and test the desired solution.
  • Developers must still test.  This is important.  Just because the agile team has a tester, doesn’t mean that a developer doesn’t have to test.
  • All testing can be automated.  Well, perhaps it could.  But it might not make sense, especially if the code isn’t stable and will need to change over sprints.  ROI, is still important within agile, so just because you can automate a test, doesn’t mean that you should.  This is the most misunderstood item that CIO’s need software quality education.

Defects

CIO’s often ask why there are so many defects found in production.  Well, that is a fairly complicated question.  In order to answer that, a full analysis will need to be done on the defects to gain a better understanding.  Many years ago, when I began a new job, the same question was asked.  In order to come up with the correct answer, the CIO brought in an outside company to perform a software testing assessment.  While I was fairly new, it wasn’t uncommon for this to occur.  In fact, I welcomed the opportunity, because I already had a hypothesis as to why this was happening.  Typically this is primarily due to little or very poor requirements.  The company came in and did the assessment and found that there were 38% of defects that were making it into production.  That is a really high percentage, most companies will average around 5%.  Over a period of time, we started to tackle the problem, and after 1 year of work, we were able to reduce the production defect leakage to 5%.  This was a tremendous accomplishment and required a team effort from project managers,business analysts, developers, and testers to make this happen.

Software Quality Metrics

CIO’s are very metrics driven.  They use data everyday to make better decisions.  While there is usually some form of metrics around software quality, it usually does not make it into the CIO’s hand for one reason or another.  I believe software quality metrics will tell a story, and provide great insight to those that are willing to look and interpret the data.  Several years ago, my team and I started to perform analysis on what data was important and which metrics would help us make decisions.  Once we agreed on what those metrics would be, we started to gather that information release over release.  We started to see trends that would help us test more thoroughly those areas which where problematic.  That resulted in bringing production defects down.  We also, built a web based dashboard, that would allow the CIO and anyone else in the company to see how testing was progressing.  Using this dashboard, we could determine if we were going to meet our testing timelines, and see what outstanding defects were holding up production deployments.  This was a true game changer for the organization.

Educating CIO’s on sofware quality will take time.  CIO’s want to have high quality software, they often don’t understand how to get there.  They don’t want their business partners to suffer through using software that doesn’t work properly.  It is important as a quality champion, you spend time with your CIO and provide software quality education, so that you can avoid having significant issues in production.  Software quality can be done effectively and efficiently within an organization.

 

Build Strong Edge Test Cases

build strong edge test casesBuild Strong Edge Test Cases

If you are a software engineer, there is a lot of effort in software testing.  With management demanding more and more quality, there is a strong push to create efficient test cases which prevent defects occurring in production.  In my 15+ years of software testing, I have found that most organizations do a very good job in covering your happy path scenarios.  I have however found that creation of negative test cases and development of edge test cases are very limited.  The main reason for this is that there is often very little time but more importantly, a lack of creativity to build these scenarios.

Creating strong edge test cases requires a very creative mind.  Sure, you need to be able to understand how the system works, but you also need to think outside the box and ask the hard questions in your test case workflows.  If you only go by a rigid set of requirements and never deviate outside of that you aren’t going to find any edge scenarios.  Building strong edge test cases requires solid application knowledge.  For example, it is important to know what will happen when the same user tries to access the same information and perform an update on it.  Will the record get locked?  Which person will update the record?  These are the types of questions that have to be answered.  Here are some additional potential edge scenarios:

  • Have a user login and disable the user to see what happens
  • Have two users try to update the same record
  • Disable the connection between the application and database
  • Have the same user try to login from two different computers

There are many more possibilities when building edge scenarios.  Over a period of time, you can begin to identify these edge test cases a lot easier, and you will begin to see tendencies which will cause these conditions.  Chances are pretty great that if the application will allow you to do something, your business user is going to try it.  These edge scenarios are also the ones that the development team is not typically going to think about, so they often will not program for it, and it will really require some thinking on their part.  These edge test cases will often stir up a lot of controversy, because these are often things that are not spelled out in the requirements.  Some of them will usually result in some significant frustration from a business perspective because it can cause a lot of uncertainty and could impact downstream processes because it wasn’t identified.

Edge test case can also typically be negative scenarios.  They could be automated, but they may not be the best candidates, because they are often complex in nature.  These are the types of things that requires deep thinking and creativity.  The testers that are the most creative, will always strive to get to the edge of coverage and push beyond in order to prevent business users from finding defects.  The good thing about edge testing is that you can perform these types of test in waterfall, agile or other SDLC cycles.

5 Step Install Robot Framework Ride using PIP

software testingIf you would like to learn how to 5 step install robot framework ride using PIP I can provide a simple process and get it installed quickly.  You have probably already installed Python, and most people use pip to make it super easy.  I have outlined the 5 steps below to install robot framework ride using pip.

 

RIDE is a lightweight and intuitive editor for Robot Framework test data.  If you would like to learn more information about the RIDE framework click here.

5 Step Install Robot Framework Ride using PIP

Step 1: Find Install location

Go to the location where you have installed Python.

Step 2: Copy the path of the folder location.

Step 3: Type cmd to open the command line

Step 4: Type cd and paste the path of the directory.

Step 5: Type pip install robotframework -ride and press Enter

5 Step Install Robot Framework Ride using PIP

That is it.  The 5 Step Install Robot Framework Ride using PIP.

 

5 Step Install PYWIN32 using PIP

software testingIf you would like to learn how to install pywin32 using pip I can provide a simple 5 step process and get it installed quickly.  You have probably already installed Python, and most people use pip to make it super easy.  I have outlined the 5 steps below to install pywin32 using pip.

 

If you have not configured Selenium with Eclipse click here.

Python extensions for Microsoft Windows Provides access to much of the Win32 API, the ability to create and use COM objects, and the Pythonwin environment.  If you want to learn more about the benefits of pywin32 click here.

5 Step Install PYWIN32 using PIP

Step 1: Find Install location

Go to the location where you have installed Python.

Step 2: Copy the path of the folder location.

Step 3: Type cmd to open the command line

Step 4: Type cd and paste the path of the directory.

Step 5: Type pip install -U pywin32 and press Enter

5 step install pywin32 using pip

 

That is it.  The 5 Step Install PYWIN32 using PIP has been installed successfully using PIP.

 

Install wxPython GUI Toolkit for Using PIP

If you are starting to learn how to use wxPython GUI Toolkit for Python so that you can get a jump start on developing test scripts using Python with Selenium.  You have probably already installed Python, and most people use pip to make it super easy.  I have outlined the steps below that are needed to get it up and running.

 

If you have not configured Selenium with Eclipse click here.

If you want to learn more about the benefits of the wxPython click here.

Install wxPython GUI Toolkit for Python

Step 1: Find Install location

Go to the location where you have installed Python.

Step 2: Copy the path of the folder location.

Step 3: Type cmd to open the command line

Step 4: Type cd and paste the path of the directory.

 

Step 5: Type pip install -U wxPython and press Enter

Install wxPython GUI Toolkit for Python

 

That is it.  The wxPython GUI Toolkit for Python has been installed successfully using pip.

If you would like to learn more about Selenium click here.

 

Install Python Robot Framework for Selenium using PIP

Install Java and Selenium WebDriverIf you are starting to learn how to use Selenium with Python it is a great idea to download a framework so that you can get a jump start on developing test scripts using Python with Selenium.  You have probably already installed Python, and most people use pip to make it super easy.  I have outlined the steps below that are needed to get it up and running.  This is the first time I have installed pip so you will be able to see the process from beginning to end.

 

If you have not configured Selenium with Eclipse click here.

If you want to learn more about the benefits of the Robot framework click here.

Install Python Robot Framework for Selenium using PIP

Step 1: Find Install location

Go to the location where you have installed Python.

Step 2: Copy the path of the folder location.

Step 3: Type cmd to open the command line

Step 4: Type cd and paste the path of the directory.

 

Step 5: Type pip install robotframework and press Enter

robot framework installed successfully

That is it.  The Python Robot Framework for Selenium has been installed successfully using pip.

If you would like to learn more about Selenium click here.

 

Learn How to Setup Selenium with Eclipse and Python

setup selenium with eclipse and python

 

Learn How to Setup Selenium with Eclipse and Python

Here are the steps needed to Learn How to Setup Selenium with Eclipse and Python

Step 1: Install Python

Go here to install the latest version of Python.

download python

Open the installer.  Click on the checkbox to add Python to path and Install Now.

install python code

Python is installing.

Install was successful.

python install successful

Step 2: Install PyDev in Eclipse IDE:

Once you are logged into Eclipse, select Help and Eclipse Marketplace

install selenium and python

The Eclipse Marketplace screen will appear.  Enter in pydev and click on search icon.

install selenium with eclipse and python

Click on the Install button.

install python with eclipse

The following window will appear.  Make sure both boxes are checked and click on the Confirm button.

confirm python installation

On the next window select the Radio button to accept the terms and click on the Finish button.

accept python terms

You will notice at the bottom of Eclipse, the python software is in the process of being installed.

install python

Once that is complete a message window will appear.  Click on Restart Now button.

restart eclipse IDE now

Once Eclipse IDE has been restarted, click on Window and Preferences

install eclipse and python

The following window will appear.  Click on PyDev.

python development

Click on PyDev > Interpreters > Python Interpreter and the following will appear:

python interpreter

click on the Config first in PATH button.  That will cause the python icon to be displayed under Python Interpreters.

Next, click on Libraries.  You will see a list of the files displayed.  Next, click on Browse for python/pypy exe

python libraries

When finished, click on the Apply and Close button.

Now, we need to create a new project in Python.  Click on File > New > Other

The following window will appear.  Expand PyDev and select on PyDev Project and click on the Next button.

The following window will appear.  Name the project NewPyDevProject and click on Finish

Next, you will need to create a new package.  On the project name right-click and select New > PyDev Package

On the next window, call your new package newPackage and click on the Finish button.

You will see the new package is created in Eclipse.

Once the new package is created, you will need to create a new module.  Right click on your project, click on New > Other

 

The following window will appear.  Click on PyDev Module and then click the Next button.

Enter in your module name and click on the Next button.

That is it.  Now you are ready to start preparing your tests in Selenium.

 

 

 

 

 

 

 

 

 

 

 

Big Changes in Selenium 4

Selenium 4Selenium 4 Alpha Download Available

If you are looking for Selenium 4 Alpha version, you can find it here.  While it is not recommended to use an Alpha version for full implementation, it will let you see what new changes are there and it will help you figure out what changes you will need to make moving forward and understand the benefits,

Selenium 4 Official Release Date

As of now, there hasn’t been an announcement on when Selenium 4 will officially be released.  Once we know that, we will provide the updated date.

Selenium 4 Changes

Here are some of the major changes that will happen in Selenium 4

  1. W3C Selenium WebDriver standardization-The goal is to have Selenium WebDriver completely standardized.  There are other tools and products that use WebDriver so this is a must have.  Specifically mobile testing tools like Appium and iOS Driver heavily rely on it.  With the change coming in Selenium 4, the test will now be able to directly communicate without using API encoding/decoding using the W3C protocol.
  2. Selenium IDE Support for Chrome-There will be much more advanced capabilities with the new update.  There will be a completely new way using a new plug-in to Selenium 4.  There will also be a new CLI Runner that will leverage NodeJS and have the following capabilities:
    • WebDriver playback
    • Parallel execution
  3. Improved Selenium Grid-While Selenium Grid is very powerful in that it allows tests to run in parallel, it is very difficult to initially setup and configure.  In Selenium Grid, the hub acts like a server and it controls which tests are run.  The node are the test machines where the Selenium test scripts are run.  With Selenium 4, there will be improved changes related to the hub.
  4. Improved Selenium Grid UI-Selenium 4 will have an improved user experince with a much richer UI.
  5. Improved Debugging/Logging capability-It is important for the automated engineers to have as much detail as possible when it comes to logging and debugging of issues.  Selenium 4 promises to provide that in order to accelerate resolution of script issues.
  6. Selenium 4 Documentation– The documentation hasn’t been updated since Selenium 2.0.  Documentation is critical and it is important that it is updated so that automated test engineers will not have to wonder how to use Selenium 4.

I hope this information has been helpful.  I will be thrilled to see the new Selenium 4 version when it is officially released.  For now, you can download Selenium 4 Alpha version from the link provided above.  Once the official version is released, you will be able to download it from the SeleniumHQ website.