Characteristics of Test Data Management

  • Introduction

Test data plays an important role in successful completion of test. Test teams not only have to follow exact test methodologies, but also ensure accuracy of data to correctly reflect production situations, both functionally and technically.

Test data scope starts with receiving the request for test data and criteria from the testing team and ends with delivering the data. This includes setting up the data environment, loading test data from production environment or creation of test data ( these vary from project to project). Test data is a critical need for Functional, Integration, System, Acceptance, Automation and Performance Testing. This depends on the agreement among the various stake holders.

A well-defined Test data management strategy can rapidly reduce inefficiencies, help extract greater value from expensive data and make validated test data available in an organized, secured, consistent and controlled manner.

The exact nature or representation of the production data needs to be understood through data profiling and discussions with business analysts. This will help understand what makes the data valuable.

Some key pieces of information shall be focused during this process:

  • Domain values: The full range of valid and meaningful values for a data field
  • Data ranges and limits: Especially those that define our equivalence classes
  • Data relationships: Data characteristics including cross- system data mappings and sources for derived or calculated data
  • Upstream and downstream: Data dependencies from upstream and downstream systems

The test data management team should understand how the above mentioned vital features are consumed by the business users.

The type of testing also impacts the test data requirement management. For example, an automation testing scenario requires highly stable, predictable data sets whereas manual testing can afford some variability. Performance tests usually require test data either represented or sampled from the production environment to replicate the real production scenario.

The following are the important activities that are performed as part of Test Data Management:

  • Initial setup of test data: This is a one time job which requires initial setup and synchronization of test data.
  • Service projects for test data requirements: Provide test data to projects based on the requests received. Projects raising data requests may be either new project which require the data to be created from scratch or can be maintenance projects where the data needs to be refreshed.
  • Continuous support to projects for test data requirements: Maintenance of test data based on requests sent by the projects. Like simple data creation request for change in data requirements, requests related to rectifying problems related to test data delivered.
  • Maintenance of data: Scheduled maintenance of test data beds in defined frequencies (weekly, monthly, quarterly or annually)

 Test Data Management Characteristics

A test data management elements are helpful not only to organize information about test data, but also to maintain this information over time.  The following data characteristics or attributes are critical to build the framework:

  • Data classification
  • Data sources
  • Data selection

Data Classification

Data classification has to be considered as an important parameter for an effective test data management.  Data classification includes the following:

  • Environmental data
  • Baseline data
  • Input data

Environmental data defines the application operational environment and is a foundational component of the test effort since it establishes our execution context. Environmental data includes:

  • System configuration
  • User authorization, authentication, and credentials
  • Configuration options

Baseline data has two fundamental purposes – to establish a meaningful starting point for testing and to establish a set of expected results. The initial baseline is set from test case pre-requisites from reliable data which helps in an ideal environment to attain the expected results. This shall help the test team reduce the efforts through automation for result evaluation process against an expected result baseline.

Input data is the data entered into the system under test to evaluate how it responds to the provided input. Observed behaviour establishes the actual results which must then be compared to expected results to determine the correctness of the behavior. Input data is typically a component of the test case itself.

Data Sources

The following are the major sources of test data:

  • Production data
  • Derivatives of production data
  • Simulated data

Production data is rarely used for testing given the risk of data security concerns and regulatory compliance requirements.   In some scenarios when it is inevitable that production data must be used for the test management the test teams should be aware of the sensitivity of the data with which they are working.

Derivatives of production data sets help to maintain the production like characteristics. A proper sanitization needs to be applied on such data to minimize the risk of security breaches.

Simulated data is useful when there is limited or no production data available. The simulated data fits to unit testing where less effort is required to create the test data.

Data Selection

All the possible scenarios need to be considered in the preparation of the test data.   Any scenarios not considered in data preparation will impact the quality of the test data used for testing.  Sample scenarios that will affect the data selection criteria are:

  • Positive and negative testing scenarios
  • Possible overlaps and redundancies
  • Statistical characteristics of the data
  • Default conditions
  • Cross-project dependencies

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Delivering Business Value Faster with BA QA Collaboration

Introduction

In today’s competitive business world, organizations enjoy significant advantage if they are quickly able to deliver software of predictable quality with minimum budgets.  But challenges like shifting customer demands and needs drive IT organizations to come out with innovative models for delivering business value faster. So the onus is on the IT organizations to implement robust and flexible QA processes that quickly adopt to changing business requirements and meet business goals. One key collaboration that can help address this challenge is the alignment of Business Analysis (BA) and Quality Assurance (QA).

BA QA Alignment

BA QA Alignment can help incorporate the agile principles efficiently and deliver needs of the business effectively. In the traditional IT model, the BA and QA teams work in silos with very limited collaboration with one another. The QA team is handicapped with lack of business knowledge and BA team with delivering quality requirements. The alignment of these two critical teams help tie the two loose ends of building the right product to start with and ensuring that the product is built right in the end.

The two main goals of this alignment are, collaborative requirements and collaborative tests.

The following steps are needed for articulating the value of BA QA Alignment:

  •  Collate – Gather business requirements
  • Communicate – Share and incorporate feedback
  • Validate – Improve the quality of requirements
  • Establish – Streamline alignment and create business value

BA QA Collaboration

In addition to the BA contribution to QA all through the test life cycle, QA can also contribute to the BA activities. Business Analysts who thoroughly understand the requirements of the system being built add a unique perspective to the testing efforts. Business Analysts in addition to confirming that the system meets functional and non-functional user expectations, can drive activities which are aimed at validating all aspects of data conversion and system deployment. Also QA can help the business analysts to write the acceptance tests for the functionality.

The following activities are where the BA and QA can support one another to deliver business value faster:

  • Define Testing Scope: QA can collaborate with business analysts to understand the application better at a functional level. It also helps QA to define the scope of testing.
  • Functional Testing: BA can support the QA team to document and test the system functions and processes of exhaustive tests to validate their functionality.
  • Requirements Validation: BA working together with QA can identify gaps in the specifications to that of features identified for development. QA can support BA by performing requirements validation.
  • Validate Data Initialization: BA can support the QA team with the right processes and utilities used to populate the system database for testing the new features. BA can give lot of valid inputs to build comprehensive test data sets and also procedures in identifying data initialization and conversion tests that will be performed.
  • Conduct Application Trainings: BA can create and distribute the user and training materials and conduct training sessions needed to familiarize the testers with the application.
  • Prepare for System Acceptance: QA can help the BA to write better acceptance tests for the functionality. QA can give valuable inputs around boundary conditions or negative testing.
  • Support Materials: QA can support BA to refine the various materials that support the use, operation, and maintenance of the system to update and reflect any necessary adjustments resulting from testing.

BA QA Alignment Benefits

  • Efficient test prioritization
  • Appropriate test approach
  • Business focused testing
  • Risk based test scenarios for test execution
  • Organized and prioritized test suite
  • Re-purposing of existing scenarios
  • Verified acceptance criteria
  • Productive defect triage discussions
  • Early defect detection
  • Defect prevention
  • Improved clarity/consistency of story cards/requirements
  • Increased quality of requirements
  • Efficient test coverage
  • Cost savings

 Conclusion

The Agile & DevOps movements influenced every team & role in an organization differently and create opportunities to learn new skills and develop new ways of working together. In today’s IT world where silos are being brought down and collaborative culture is the success mantra, the alignment of the two very critical components of IT organization each adding their own perspective, certainly gives a great value add to deliver quality faster and better. The BA QA Alignment creates a platform where individuals with right skills are brought together to drive efforts on building the right product at the outset and ensuring they did it right upon completion.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Regulatory Compliance & DevOps

Introduction

In today’s IT world the need for organizations to protect sensitive information of their customers is more than ever. Governments around the world have enforced policies to protect companies against poor management of sensitive information. Compliance is one the very critical components in IT. But in IT this has a negative inference. The reason being it brings too much of documented processes involving discussions, paperwork which in turn slows down the more important work of releasing the software.

As the trend of implementing DevOps in IT organizations continue there is a need to understand how this will affect security and compliance controls. These controls are key enablers of software development activities. So it is very important for organizations to think how regulatory compliance fit in with the cultural and organizational shift which is brought by DevOps to support continuous delivery of software.

Regulatory Compliance in DevOps World

DevOps advocates delivering software in agile fashion by automating the IT delivery chain. This appears to contradict the regulatory goal of compliance to ensure that the organization isn’t opening itself up to potential vulnerabilities. Regulatory compliance are mandatory in regulated industries like Healthcare, Banking and Finance.

DevOps automation techniques for continuous delivery can appear to dissatisfy regulatory compliance. But the fact is that DevOps automation can help organizations not only to stay compliant, but actually increase their compliance levels. DevOps teams must not only include compliance activities early in the software life cycle as in the case of testing but also automate the compliance tests. Instead of leaving the security and compliance concerns for later in the release cycle it is better to deal with them early in the development life cycle. By doing so it will be easy to remove the compliance bottleneck early and bring security, quality, agility and stability into the software value chain. Automation of processes and tests will help reduce the risk of introducing security flaws due to human error.

Maintaining audit trails of software development activities is the key requirement of regulatory compliance. The continuous integration mechanism in DevOps can log and track precisely what version of each source code file contributed to which version of the software. Also with continuous deployments each build can be tagged to assure that the deployments are inspected to deny unauthorized changes moving forward. The automation testing of every build can be checked for regulatory issues. Automation of infrastructure provisioning is another area of DevOps that will help satisfy the regulatory compliance. Infrastructure provisioning scripts as verifiable and testable which can be reliable and reproduce results. All these satisfy the mandatory regulatory compliance.

By adopting the DevOps automation techniques which extend the entire software delivery pipeline, the ability to control, audit, and protect the organizational assets will only increase. This will ensure that the organizations are compliant with regulatory requirements.

Conclusion

DevOps guiding principles like automation and validation actually provide in depth audit and change information to satisfy audit and regulatory compliance needs. Another compliance concern which is governance is also addressed in DevOps by advocating processes for creating, communicating, and enforcing policies which also includes security and compliance policies across an organization. DevOps actually complements existing processes and methodologies such as ITIL & Agile which help organizations to be compliant.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

QA challenges in DevOps

Introduction

Quality Assurance has always been an evolving discipline in software development. With the emerging trends in IT industry, the need to better understand, manage, and adopt the QA activities is increasing. With the onset of agile and lately DevOps, the way organizations develop software has changed, and so have the ways to enforce QA. Software development cycles have become short and quick. With this QA teams face new challenges as they work to keep pace. The advantages of overcoming the challenges include quality, optimization, process improvement and higher productivity.

 

Understanding QA in DevOps Landscape

DevOps advocates good principles and practices that help improve communication and collaboration between the organizational silos. This also implies to QA organization and their development counterparts. But in a DevOps scenario, the walls will be eliminated and this helps facilitate knowledge sharing, experience and specialized skills to deliver quality systems. In the era of DevOps the focus of QA teams will be more on preventing defects than finding them.

 

Challenges faced by QA teams

QA culture – In the context of DevOps, quality requires a change in how it is being conducted. This also implies an intense transference in the organizational culture as well. It is very important and also challenging to think of innovative ways of identifying unique techniques to test the software quickly and efficiently. This will enable to continuously ensure quality while also growing and evolving the QA services provided.

Facilitation of quality – From a DevOps perspective, QA team needs to understand the business for the system being verified. For this to happen, QA team should partner with business experts, including the product owner, to understand how the system being tested needs to function in order to support the business. QA teams will be disabled if not involved in those initial discussions. This involvement helps QA to become the facilitator of quality.

Collaboration – QA is the binding entity between development and operations. So QA team should be involved right from the early stages of development. This will enable them to collaborate to have software developed and supported more effectively. Also QA should be considered as responsibility of entire project team rather than the responsibility of dedicated QA team.

Early testing – One of the main objectives of testing in DevOps is early detection of defects in development cycle. For this to happen testing must begin very early in the cycle. QA should begin testing with whatever code is available even if the features are not complete. This requires lot of maturity in documenting self-sufficient user stories that do not depend on another for testing.

Test coverage – In DevOps there is a rush to deliver software quickly with the techniques like continuous integration and deployment. Also because of rapidly changing requirements, there is a possibility to miss testing critical functions.   To overcome this challenge, a thorough and detailed traceability of requirements to test functions should be maintained.

Build verification – As DevOps encourages frequent builds, there is a higher possibility of code breaking existing features.  For this reason it is not practical to have testers do these verifications manually. It is recommended to rely on automated testing for these smoke tests.

 

Conclusion

If the above discussed challenges are addressed then QA in DevOps can play a critical role in accelerating development and release schedules. DevOps guiding principles like test first, free communication and seamless collaboration help resolve some of the QA challenges and also enable the QA team to take their deliverables to the next level. In DevOps testing is a continuous process and supports the process of incorporating continuous feedback to enable better quality.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Measuring DevOps Success

Introduction

New age IT world is making revolutionary advances in ways nobody predicted. Developing applications such as the cloud, advanced analytics and expanding mobility have transformed the vision of software industry. In today’s IT world it is all about agility with quality. Traditional processes and methods of developing software are challenged with keeping up with the complexities that come with these new environments. As a result, DevOps has come to the fore as a new IT philosophy which will help overcome these complexities. The aim of DevOps is to bring collaboration among the different IT teams. DevOps is successful in increasing the agility and in faster software/application deployment. But there is no success without measurement. So DevOps success also needs to be measured.

Measurements

Measuring the success of DevOps is very important because DevOps initiatives target the very processes that determine how IT works. A metrics-oriented mindset is key to ensure that DevOps initiatives deliver the intended results. Data-driven decisions and focused improvement activities lead to increased quality and efficiency. Also the use of feedback to accelerate delivery makes DevOps a successful IT culture. With DevOps, as with any IT initiative, knowing what to measure is always the first step.

The following suggestions help identify the key measures.

Measuring Culture:

This is a tough area to create the mechanism to measure. Because it is very difficult to assign a dollar value. DevOps advocates eliminating stress, creating an environment of collaboration. This enable the people to work more effectively. Some key representative metrics are:

  • Staff retention
  • Change acceptance
  • Resource cross skilling
  • Knowledge sharing
  • Resource utilization

Measuring Agility:

Agility again is not clearly defined. One should define the measurable attributes of agility. Some of the reasons why the DevOps is fast gaining acceptance are the ability to deliver software faster, with fewer defects and faster resolution of problems. Some key representative metrics are:

  • Speed of deployments
  • Time it takes to fix failed releases
  • Frequency of releases
  • Change lead time
  • Sprint velocity

Measuring Quality:

The success of DevOps is directly related to the value and the quality delivered to the end users. The key objective of any DevOps practice should be to contribute creatively and improve the existing practices. While measuring quality using metrics one can evaluate, modify, and improve processes over time. Some key representative metrics are:

  • Production outages
  • Defect rate in production
  • Mean time to recovery
  • Deployment roll backs
  • Success rate of deployments

 Conclusion

As Lord Kelvin (Scottish mathematician and physicist) said, “Without measurement there can be no measurable improvement”. DevOps is evolved from a rich history of process improvements. Feedback and measurement are the factors that drives the improvements in DevOps. The success of DevOps is to increase business value by making it agile through continuous delivery of services that satisfy customer needs. There are several tangible and intangible benefits of DevOps. They require measurement to help an organization better appreciate the impact of a DevOps approach. If this goal is not met then whatever is being done is not working or is not right.