Continuous test improvements in the age of DevOps

Introduction

Lately there is a huge traction towards DevOps in the IT world. Many companies have adopted agile processes in pursuit of faster releases. So the logical next step to this is DevOps. So now every other organization wants to jump onto the DevOps band wagon. If we study the DevOps landscape, testing is a critical part of the handoff between development and IT operations. So testing should be continually improved for the success of agile teams using DevOps practices.

Continuous testing in DevOps

Every one of us who understand DevOps are familiar with Continuous Integration and Continuous Delivery. But the most important bridge that keeps these two entities to work effectively and which makes the concept of DevOps successful is ‘Continuous Testing’. It is not just automation as most people think. Continuous Testing goes beyond automation and encompasses all practices — including tooling and cultural change. It is mechanism of continuous validation that keeps the DevOps engine moving forward. It provides continuous feedback to the project teams to guide them towards successful software implementation.

Continuous testing in the context of DevOps should take a test-first approach, which will help measure test coverage to check that software features work as expected. It will enable to find if changes break existing test suites. And also help the project teams to focus on what they need to deliver.

As part of continuous testing, test teams can also benefit with ‘Exploratory Testing’ by simultaneously understanding the system, designing the tests and executing the tests. This will add value to the test automation by actively exploring the system without a script to learn how the system behaves in unanticipated ways.

Continuous test improvements

Test improvements are a critical part of continuous testing. They drive the continuous effort to optimize test performance of the software. This should and will provide improvements through automation and reuse of best practices. The continuous test improvement can set thresholds for each test phase that determine whether  failures  are significant enough to revert the changes or it is necessary to stop the DevOps cycles long enough to fix the problems. Test first driven development, behavior driven development concepts allow test processes to improve in a DevOps Model. Some of the key areas that can be addressed as part of continuous test improvements are code analysis and test optimization & virtualization.

While test automation acts as a driver for detecting software defects, the concept of continuous testing highlights process improvements that can prevent future defects from occurring.   Test improvements enhance real-time assessments and continuous measurements to refine the development process so that business expectations are continuously met.

Conclusion

Continuous testing is a great complement to the DevOps movement. Organizations cannot be successful with DevOps if they cannot improve the existing processes. They should know how and what to improve. Continuous testing which acts as an enabler to test improvements can be a good medium through which success is achieved. What is more important is that the ideas of test improvement should be embedded into the DevOps culture.  As I said in my earlier posts, quality is built into the entire system with DevOps. When implemented the right way, the concept of continuous testing in DevOps will be more than a method of enforcing test automation tools or best practices.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

How DevOps Will Transform Software Testing

Introduction

Aristotle said, “Quality is not an act, it is habit. What we can interpret is that quality cannot be a response to a certain situation or something that is faked or postured. Rather quality should be part of a routine; the acts we do on a continuous basis and which should be a reflection of the essential nature of work. This is how today’s IT world is looking at the QA practice.

In a traditional IT world, testing is often seen as separate from the development. They were two distinct entities separated by ideological walls. Thanks to the new trends in IT world, these walls are fast disappearing. One of the major driver for this transformation is “DevOps”.

DevOps

So what is DevOps? It is not a methodology or a suite of tools as many people think. DevOps is a culture that enables seamless coordination among the different stakeholders in the software chain. DevOps brings together the two critical silos of software process namely, development & operations to integrate within the same software cycle.

DevOps drives all the IT stakeholders to collaborate to deliver value faster. For this it collates the practices of continuous integration, continuous testing, continuous deployment and continuous monitoring.

DevTestOps

As we know DevOps aims to break the barriers between Dev and Ops. It also demands breaking the barriers between Dev, Test and Ops. Testing is a critical part of the handoff between development and IT operations. For DevOps to be successful it is mandatory that testing should be fully integrated into the software development and delivery.

With this mindset, everything changes for a traditional testing organization. Testing operations undergo a major changes in a DevOps culture. This mandates a shift in tester’s skillset, methods, tools, timing and approach. DevOps advocates starting test early and constantly.

Some tips for successful DevOps testing:

  • Involve development team during test design.
  • Identify the critical test cases and prepare the test suite to test the functionality of the required build.
  • All the environments required for testing need to be standardized and deployments have to be automated.
  • Adopted test-first approaches such as test-driven development (TDD) and behavior-driven development (BDD).
  • Keep the test execution cycles short and quick.
  • Adopt regression test automation to run across various environments.
  • Implement code analysis and coverage tools to ensure 100% code coverage.
  • Critical bugs found need to be reported, fixed and passed through the same chain of events before the code is deployed on to the production environment.
  • Encourage parallel execution of tests to help reduce time for go live which is the heart of a successful DevOps implementation.

Testing maturity is a key differentiator for the success of DevOps. Even if organizations can automate their integrations, builds and delivery processes but they still struggle to manage test orchestration and automation. Testing brains play a critical role to achieve this with their expertise in test design, test automation and test case development with DevOps.   Irrespective of what DevOps processes, models and tools organizations use, testing is a vital part of the overall DevOps process — not only to ensure code changes work as expected and integrate well — but to ensure the requirement changes do break the functionality.

DevOps Testing Tools

In order to realize the desired business goals of DevOps, it is essential to have right tools to maintain the quality associated with the delivery chain. This calls for accurate and comprehensive testing. So it becomes very important to understand the existing testing process and identify the right testing tools.

Some recommendations of testing tools are below:

  • Selenium is a portable software testing framework for web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting language (Selenium IDE). It also provides a test domain-specific language (Selenese) to write tests in a number of popular programming languages, including Java, C#, Groovy, Perl, PHP, Python and Ruby.

 

  • XL TestView is a test management and analysis tool that allows you to define and execute tests across your full spectrum of test tool suite. It analyzes test results across multiple test tools and track release metrics and quality trends over time. It helps aggregate and visualize results to provide true insight into quality.

 

  • Cucumber is a testing tool that computer programmers use for testing. It runs automated acceptance tests written in a behavior-driven development (BDD) style. Cucumber is written in the Ruby programming language. Cucumber projects are available for other platforms beyond Ruby.

 

  • FitNesse is a web server, a wiki, and an automated testing tool. It is based on Ward Cunningham’s Framework for Integrated Test. FitNesse is designed to support acceptance testing rather than unit testing in that it facilitates detailed readable description of system function.

 

Conclusion

As it is said, “Nothing is permanent but change”. Waterfall model gave way to iterative model which in turn was replaced by agile as the preferred choice for software development. DevOps is the future. DevOps isn’t an individual effort, it’s a core value of IT organization. The only reason that DevOps works is because quality is built into the entire system. So it’s time we embrace and adopt this change.

 

A special thank you to Moin Syed for writing this article.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Exploratory Testing within Agile

It is very hard to explain something that we already do. Exploratory testing is fast catching up with testing community in the context of agile. Exploratory testing is a form of ad hoc testing. But adhoc testing is considered to be more unstructured and does not get a thumbs up with the project teams.

As a software tester, one will perform exploratory testing at some level in day today activities. So to give it a more structure accountability, one needs to set a right frame work. The objective behind the exploratory testing should not just be to perform it but to find better ways to execute and to think to what level it should be performed.

Exploratory testing fits in nicely in the context of agile testing. We can define exploratory testing as a progressive approach where the testers have to perform multiple things simultaneously; understand the application functionality, develop test design and execute tests. Exploratory testing will be effective if it is based on risk-based testing and requirements-based testing. Exploratory testing can help software testers to keep up with fast paced development of agile software projects.

The benefit of exploratory testing is that it is not just driven by the documented steps but evolves with the testers’ imagination and insight into the application. As long as the tester has the skills to listen, understand, think and report accurately and effectively, exploratory testing can be very productive and produce results not anticipated by a document.

Now to fit exploratory testing in agile context it cannot be adhoc. It needs to be time boxed to align with agile structure. Boundaries of the exploratory testing should be defined within the sprint time box. The black and white box testing techniques may be used along with exploratory testing to make it more effective. This ability to of the tester to react and adapt to change quickly makes exploratory testing a very strong component in agile.

A practical method that defines the set of heuristics should be applied when testing which will guide the tester to perform exploratory testing. The below categories will help create a framework of sort to define and conduct exploratory testing:

  • Landscape: what the product is
  • Functionality:  what the product does
  • Data Flow: what it processes
  • Platform: what it runs on
  • Operations: Who will use

Exploratory testing will make agile testing more result driven. It is an efficient and effective method to test software. It gives the much needed results immediately and finds the critical defects faster. Slowly it is becoming an industry standard for manual testing.

A special thank you to Moin Syed for writing this article.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.

Exploratory Testing Framework

Introduction

Some of the common complaints about exploratory testing are that it is it needs subject matter expertise, it is not structured and it cannot be measured. But the fact is that exploratory testing is self-managed and self-structured. Exploratory testing is not a technique but an approach of exploring software without following documented steps.

Skills needed for exploratory testing

Exploratory testing can be more of skill based testing than a role based testing. Some of skills like questioning, observation, critical & lateral thinking will help an exploratory tester explore better. The ability to think beyond just a one sentence requirement and related test steps is part of the inherent skill which is a core need for exploratory testers. A well-executed exploratory testing rely on skill of the person. The tester uses his skills to approach the problem and controls the process. The tester continuously implements the feedback received from the previous test. So we can say that, exploratory testing is highly interactive and is an approach that allows in creating test designs when new information is discovered.

Structured exploratory testing

Some of the best practices during exploratory testing are to define test charters, test design heuristics and time boxing. These techniques will put some general guidelines around the testing being done and it gives a structure to define its success.

Test charters

Test charters give exploratory testing sessions a mission without being authoritarian.

Test charters outline:

  • What areas of the system needs to be concentrated which is the mission scope
  • When is the application ready to be used
  • Responsibilities of the team doing the exploratory testing
  • Duration of testing sessions
  • The date and times of the testing
  • The test environment being used to do the test
  • Test data needed for testing

As the exploratory testing session progresses, charter should be updated to document observations or issues the team come across which can be used for further testing.

Test design heuristics

In today’s IT world where the functional complexities are always increasing and requirements always changing which makes launch applications to market even more challenging. It becomes imperative to think scientifically, use questions, explain the behavior, and based on the experiences gained from testing user predictions to create a test strategy. Scientific approach and using heuristics give the structure to exploratory testing.

The purpose of heuristics is to remind testers of what to think about when they are creating tests. Following are some pointers:

  • Resources, constraints, and other elements in the project that may support or roadblock testing. Sometimes a tester must challenge constraints, and sometimes accept them.
  • Software functionality is complex. All the corner cases to test should be covered. Cover all of it that matters, not just the parts that are easy to see.
  • Quality criteria needs to be adopted that enables to determine if the application has problems.
  • Test techniques are heuristics for creating tests. All techniques involve some sort of analysis of project environment, functional elements, and quality criteria.
  • Observed quality is the result of testing. One can never know the “actual” quality of a software product, but through the application of a variety of tests, one can make an informed assessment of it.

Time boxing

Exploratory test session can be limited to a set timescale, a practice also known as time boxing. This will be a focused test effort of fixed duration. This can help concentrate on the project’s specific goals and scope, rather than drift into unfocused exploration.

Measuring efficiency

The efficiency of the exploratory testing can be measured by the time spent on the following:

  • Test design and execution (T)
  • Defect investigation & reporting (D)
  • Configuration (C)

Analyze on the basis of the following criteria:

 

  • More of ‘T’ indicates great code but might also mean poor defect identifying skills.
  • More of ‘D’ might mean poor code quality but also suggest inefficient test reporting.
  • More of ‘C’ reflects configuration and testability issues but it can also mean system is not ready to test yet.

The above method of measure learned from other practitioners is very helpful in calculating the efficiency.

Value of exploratory testing

The value of exploratory testing over other testing methods is

  • Identifies complex defects in a system earlier
  • Facilitates experimentation, discovery and learning
  • Creates engagement by enabling people use their minds
  • Provides user-oriented feedback to developers and business analysts

A special thank you to Moin Syed for writing this article.

If you would like more information on Agile, DevOps or Software Testing, please visit my Software Testing Blog or my Software Testing YouTube Channel.