When Test Automation ROI Doesn’t Make Financial Sense

test automation ROI

There are several situations where test automation doesn’t make good financial sense. Let’s face it, test automation is great, and I am the greatest champion of test automation. If done effectively, it will result in tremendous Test Automation ROI and significantly increase the ability for the QA team to test a wide assortment of tests. The key is you have to be smart. Just because you can automate it, doesn’t mean that you should.

QA has become highly technical. The technical capabilities of teams today are far more advanced than they were 5 years ago. This has tremendous advantages but it can also be extremely dangerous. The same resources who can work circles around their predecessors are also the ones that are laser focused on proving their technical acumen and aim to prove their worth to their bosses and peers. One thing they forget: Test Automation ROI.

Test Automation requires resources and those resources cost money. All work should be closely monitored and justified. It has to make good financial sense and it should provide a high Test Automation ROI within a short period of time. The challenge is that when technology meets business the technical does not always translate to justification of automation.

I have been using automation since I started my testing career 20+ years ago. I have learned a lot and I have both automated and had teams that automated things that other previous teams had miserably failed. I constantly heard statements like : “you can’t automate that”, “we have already tried that”, “I can execute it faster than a script”, “this is a waste of time”. I have proved all them wrong.

I have seen a few situations where test automation simply doesn’t make sense. I have made these mistakes and I have learned.

  • New Applications: It doesn’t make sense to automate brand new applications right from the start. With new applications, requirements are constantly changing and the UI goes through many iterations. Not only is this difficult to test from a manual perspective, it becomes impossible to keep up with the automated scripts. The best thing to do, is to give it time and come back once things have stabilized. You will be able to automate scripts a lot faster, and it will result in less frustration and a higher ROI.
  • Frequent UI changes: If you are creating test automation scripts and the UI is constantly in flux, it will result in constant script updates and it often results in things becoming unmanageable. Every code build will result in automation scripts which are broken, and those will have to be updated. My best advice is to be patient and wait.
  • Data Model updates: If your application is going through a lot of data model changes, it will result in UI changes which will have an impact on your scripts. These data model changes are often done early on, so if you give it sometime to settle, you can step in and automate those components.
  • Stable Code: One of the most important factors is to have stable code. If the code isn’t stable, and there are a lot of defects and frequent code builds, this will often result in a high degree of test automation failures. It is crucial to let the code settle, and in some situations have less code delivery, in order to maximize test automation coverage. More builds don’t necessarily mean better applications.

I hope this information has been helpful. I wish you the best as you work to build smart test automation. And I hope you build tremendous Test Automation ROI!

What is the Cucumber Testing Tool?

what is the cucumber testing tool? Cucumber is one of the most popular tools for behavior driven development testing.  Cucumber enables the creation of automated behavior driven development acceptance tests leveraging the Gherkin framework.  Cucumber is open-source so you don’t need to worry about paying expensive licensing fees.  Cucumber testing tool is written in the Ruby programming language.

What is Behavior Driven Development?

Behavior-Driven Development (BDD) is a software development process within Agile that Cucumber was designed to support.  It is one of the leading BDD testing tools on the market today.

BDD enables:

  • Open communication and collaboration across roles
  • Work in small iterations increasing feedback and creating business value
  • Producing documentation that enables automation of user acceptance tests

BDD does not replace the Agile process but rather enhances the overall process.

Cucumber Testing Tool Benefits

  • Testers will be able to write automated tests without needing in depth understanding of a programming language
  • Cucumber supports multiple programming languages
  • There is a high degree of reusable code
  • Cucumber is quick and easy to setup
  • Cucumber allows integration with Selenium, Watir, Spring, Ruby on Rails and many more

Cucumber Installation

If you are looking for information on how to install Cucumber you can go to the main website located here.  Cucumber is widely used so there are many programming languages across multiple platforms.  It is always a good idea to install the one that closely matches your application.  Some of the most common programming languages used with Cucumber are: Java, Ruby, PHP, Python, and Perl.

What is Gherkin Framework for Behavior Driven Development?

using Gherkin framework for BDDBehavior Driven Development has gained a lot of traction over the past several years with the adoption of Agile across most IT organizations.  One of the most common used tools is Cucumber and it uses the Gherkin framework to automate tests.  Gherkin language is defined as a business readable, domain specific language created especially for behavior descriptions.  The purpose of Gherkin is to promote behavior driven development across the entire team.  This includes business analysts, developers, testers, and product owners.  The Gherkin framework promotes firmly written requirements and eliminate ambiguity.  While Gherkin is primarily used in English, there are a total of 37 spoken languages that can be used.  Gherkin serves two basic purposes: provides project documentation and helps in creating automated tests.  The most commonly used automation tool for Behavior Driven Development is Cucumber.

Gherkin Language Keywords

Here are the primary Gherkin keywords:

  • Feature
  • Rule
  • Example or Scenario
  • Given, When, Then, And, But
  • Background
  • Scenario Outline or Scenario Template
  • Examples

Here are secondary Gherkin keywords:

  • Document Strings: ” “
  • Data Tables: |
  • Tags: @
  • Comments: #

Gherkin Syntax Example

#This is an example of how you can use the Gherkin syntax

Feature: Login functionality of Amazon

Given: I am an Amazon user

When: I enter username as username.

And I enter the password as password

Then I should be redirected to the home page of Amazon

Gherkin Best Practices

  • Each feature should be executed separately
  • Each scenario should be executed separately
  • Leverage your scenarios with your requirements
  • Make your steps modular and simple to understand
  • Combine common scenarios

Gherkin Advantages

  • Gherkin syntax is easy and simple for everyone including non-programmers to understand
  • Programmers can leverage Gherkin to write automated tests
  • Requirements are firm and unambiguous
  • Gherkin can be used to write user stories
  • You don’t need to be an expert to understand Gherkin
  • Gherkin automatically links acceptance tests to automated tests
  • There is significant reuse of tests

Gherkin Disadvantages

  • Requires a lot of collaboration across the team and business partners
  • There might be some situations where it isn’t an ideal solution
  • Poorly written Gherkin will result in high test maintenance

What is Test Driven Development (TDD)?

what is test driven developmentThere has been a lot of information shared over the past few years about Test Driven Development.  Initially it might seem complicated and counter to logical thinking but it actually makes a lot of sense.  Test Driven Development is defined as: a software development process that relies on the repetition of a very short development cycle.  Requirements are turned into very specific test cases and the software is modified in order to get those tests to pass.  In the normal process of software development, code is written first and then tested to make sure that it works.

 

 

 

 

What is the origin of Test Driven Development?

The origin of TDD traces back to the late 1990’s when extreme programming started.  It because more mainstream when Agile came around.  By 2006 it had matured and resulted in additional innovations to the process with ATDD or BDD.

What are the steps for Test Driven Development?

Here are the basis steps for TDD:

  1. Write a new test
  2. Check and make sure the test fails
  3. Write code to pass the test
  4. Verify the test passes
  5. Refactor code to make it more efficient
  6. Repeat steps again

What is the purpose of Test Driven Development?

TDD is a process of modifying code in order to pass a test which was designed prior to code being written.  There is more emphasis on writing production code than designing test cases.  This includes refactoring and helps to have cleaner and simpler code.  It helps to find defects much sooner and eliminates the need of rework.

What is Agile Test Driven Development?

All the teams today that use Test Driven Development follow Agile or some form of it like Scrum.

Who is responsible for Test Driven Development?

Typically the person who is writing the code will be the person to write the test cases.  I have seen some situations where there is a slight variation and a test engineer writes them but that is not a common practice.

What are the benefits of Test Driven Development?

  1.  The primary benefit of TDD is that it helps to eliminate writing duplication of code
  2. Allows developers to capture defects much sooner in the lifecycle
  3. Software is better designed and results in more maintainable code
  4. Helps with teamwork.  More people can work on code together.
  5. Less time is spending on debugging code
  6. Helps with code refactoring
  7. Eliminates a significant number of defects
  8. Often results in projects getting completed faster than original timelines

What are the pitfalls of Test Driven Development?

Here are some common pitfalls of TDD:

  1. Forgetting to run the tests
  2. Writing too many tests
  3. Writing tests that are too large
  4. Writing tests that are too simple
  5. Writing tests that are too complicated

Test Driven Development Skill Levels

There are different skill levels for TDD which include:

  • Beginner
    • Someone who can write a unit test prior to code creation
    • Someone who can get a failing test to pass by writing code
  • Intermediate
    • Someone who can write a test case for a defect that has been found
    • Someone who can take a previous test driven development test and further break it down to simpler tests
    • Someone who can factor reusable elements
  • Expert
    • someone who can create a test driven development roadmap
    • someone who can provide additional guidance and direction to beginner or intermediate levels

I hope this information has been helpful.  You might want to also learn more about Behavior Driven Development.

What is Behavior Driven Development (BDD) Testing?

behavior driven developmentThere has been a lot of information published about Behavior Driven Development or BDD.  It is counter to what most people are used to so it causes a lot of confusion.  It is very popular and a lot of organizations have started to embrace this methodology which results in great success to those who can consistently apply some basic principles.  Behavior Driven Development allows companies to shift further left and identify issues much earlier in the process than using the traditional waterfall methodology.

What is BDD?

Behavior-Driven Development is an extension of Test-Driven Development or TDD.  Behavior-Driven Development is an approach to build features based upon user stories.  You will typically have a product owner that will communicate the expectations for the software in form of a user story.  That information will be stated in terms of business objectives or goals.  Both the developer and the tester will use this information to both develop and test if these features meet the expectation of the product owner.  While BDD is a process, just like any process it requires great communication between all members.  What does the product owner want to see?  How can that be translated into software features?  Are those features really needed or are there other features that might meet business needs.  Behavior-Driven Development heavily relies on communication and collaboration which are also one of the key tenants of Agile.

BDD Testing

Behavior Driven Development premise is that the tests are written before the code is developed.  In basic terms it tells you how a piece of code needs to be tested.  You want to test the behavior of a given feature.  It is extremely important to do this first so that you will have a high degree of testing coverage.  BDD requires the person who is creating the tests to think about the business scenario.  As you build code, you are building a very large repository of tests which can be executed over and over again using tools such as Jenkins.  Now you might wonder if the developers are writing all these tests, why are testers needed?  Well testers are needed more than ever.  Developers are often focused on a small module or piece of code and don’t have an overall understanding of how the system works as a whole.  Testers typically have a broader understanding and are usually business subject matter experts.  It is important for testers to understand the broader context of the what and why of software and the business intended use.

Three Best Practices of BDD

  1. Discover: The first best practice is the most important one in my opinion.  Created a shared understanding across the business and the Agile team of what the requirements are through collaboration.  This is a critical step, and one that most Agile teams will overlook and rush to build tests and code.  This collaboration needs to occur through structured conversations using specific rules and examples.
  2. Define:  The second best practice is to use real world business scenarios and document how the system should behave.  This documentation reinforces best practice 1 and 3.  The most commonly used framework for defining scenarios is Gerkin.
  3. Automation:  The third best practice is to automate the documentation.  This will allow the documentation to grow and become dynamic.  This process will verify that the system works as expected and verifies best practice 1 and 2.  Most teams today use Cucumber to automate BDD tests.

BDD Framework Process

Here is a sample flow of how things work within the BDD framework:

  1. Create a user story with high level functionality
  2. Hold a requirements session and further define functionality with business examples
  3. Define business scenario using Gerkin
  4. Automate the scenario using Cucumber
  5. Write code so that the test scenario will pass
  6. Run additional tests including regression, performance, etc.
  7. Release code into production

I hope this information has been helpful and has provided you with some great information about Behavior Driven Development