Measuring DevOps Success


New age IT world is making revolutionary advances in ways nobody predicted. Developing applications such as the cloud, advanced analytics and expanding mobility have transformed the vision of software industry. In today’s IT world it is all about agility with quality. Traditional processes and methods of developing software are challenged with keeping up with the complexities that come with these new environments. As a result, DevOps has come to the fore as a new IT philosophy which will help overcome these complexities. The aim of DevOps is to bring collaboration among the different IT teams. DevOps is successful in increasing the agility and in faster software/application deployment. But there is no success without measurement. So DevOps success also needs to be measured.


Measuring the success of DevOps is very important because DevOps initiatives target the very processes that determine how IT works. A metrics-oriented mindset is key to ensure that DevOps initiatives deliver the intended results. Data-driven decisions and focused improvement activities lead to increased quality and efficiency. Also the use of feedback to accelerate delivery makes DevOps a successful IT culture. With DevOps, as with any IT initiative, knowing what to measure is always the first step.

The following suggestions help identify the key measures.

Measuring Culture:

This is a tough area to create the mechanism to measure. Because it is very difficult to assign a dollar value. DevOps advocates eliminating stress, creating an environment of collaboration. This enable the people to work more effectively. Some key representative metrics are:

  • Staff retention
  • Change acceptance
  • Resource cross skilling
  • Knowledge sharing
  • Resource utilization

Measuring Agility:

Agility again is not clearly defined. One should define the measurable attributes of agility. Some of the reasons why the DevOps is fast gaining acceptance are the ability to deliver software faster, with fewer defects and faster resolution of problems. Some key representative metrics are:

  • Speed of deployments
  • Time it takes to fix failed releases
  • Frequency of releases
  • Change lead time
  • Sprint velocity

Measuring Quality:

The success of DevOps is directly related to the value and the quality delivered to the end users. The key objective of any DevOps practice should be to contribute creatively and improve the existing practices. While measuring quality using metrics one can evaluate, modify, and improve processes over time. Some key representative metrics are:

  • Production outages
  • Defect rate in production
  • Mean time to recovery
  • Deployment roll backs
  • Success rate of deployments


As Lord Kelvin (Scottish mathematician and physicist) said, “Without measurement there can be no measurable improvement”. DevOps is evolved from a rich history of process improvements. Feedback and measurement are the factors that drives the improvements in DevOps. The success of DevOps is to increase business value by making it agile through continuous delivery of services that satisfy customer needs. There are several tangible and intangible benefits of DevOps. They require measurement to help an organization better appreciate the impact of a DevOps approach. If this goal is not met then whatever is being done is not working or is not right.

QA Engineer Job Description

I have recently written a QA Engineer job description and I though I would share it.

The Quality Assurance (QA) Engineer I is primarily responsible for testing all software products—built in-house, purchased and installed locally, or configured in the cloud. The  QA Engineer I partners with internal
stakeholders to plan, test, and implement all software changes to ensure that
products conform to the desired quality standard.  This role requires technical skills and the aptitude required to identify and resolve critical issues. The QA Engineer I uses technical programming expertise to build automated and performance tests.  The individual will be expected to contribute
towards the organizational goal of continuous process improvement.  This role requires strong communication and the ability to work well with others.

Responsibilities include:

  • Develop automated testing strategy or performance testing strategy
  • Prioritize automated test scripts or performance scripts and executes according to plan
  • Develop automated test scripts to automate manual repeatable test cases
  • Create and maintain automated or performance scripts using C++, JAVA, and/or VBScript
  • Create new test data or leverage existing test data for automation or performance scripts
  • Identify, log, and track defects to closure
  • Provide reporting on defect tracking and escalate technical issues as needed
  • Develop and maintain testing standards, procedures, and guidelines to ensure consistent testing procedures
  • Partner with distributed teams, outside vendors or offshore testing partners
  • Follow standards in accordance with company policy and regulation (SOX, Lean, Six Sigma, etc.)

II. Know-How is the sum total of every kind of skill, however acquired, necessary for acceptable job performance

A.              Technical/Specialized:

  • Working knowledge of HP QTP and/or HP LoadRunner required
  • Ability to program with VB, JAVA, and/or C++
  • Ability to write and execute SQL to retrieve data from databases
  • Ability to anticipate user behavior or identify risks to systems
  • Solid understanding of automation and performance testing concepts
  • Broad technical knowledge, including hands-on experience, managing system testing in new computing architectures/environments. Knowledge of relevant tools, databases, and middleware
  • Working knowledge of system performance including CPU and memory utilization.
  • Working knowledge of automated testing methodologies, including selection, configuration, and maintenance of modern automation tools
  • Understanding of basic waterfall and agile/iterative project management concepts.

B.  Managerial:

  • Highly focused on building quality assurance best practices
  • Ability to influence change in order to improve processes and application quality.
  • Comfortable working with dispersed/offshore teams.
  • Able to convey calm but focused sense of operational urgency to team members.
  • Assists in implementing changes in order to increase operational efficiency. 

C.  Human Relations Skills:

  • Strong customer service focus
  • Strong verbal and written communication skills. This role will regularly explain complex issues to diverse stakeholders, including top management. At times, this role will need to train non-technical business users in QA testing processes and methodology. 

III. Problem Solving:

  • Strong problem solving and analytical skills. This role will be required to creatively use, analyze, and improve numerous new processes.
  • Strong analytical, problem-solving, and conceptual skills. This role will be required to assist in developing clear solutions to complex problems, often where no current solution exists.

Education and Experience:

  • Bachelor’s in computer science or equivalent
  • 1-3 years programming experience with C++, JAVA, VB, and SQL
  • 1-3 years experience testing applications in accordance to required functionality and non-functional quality attributes (e.g. availability, performance)
  • IT testing certification (CSTE, CSQA) is a plus.
  • Master’s degree in computer science is a plus.
  • Experience writing automated or performance test plans
  • Experience writing automated or performance scripts
  • Solid understanding of the IT project and software development lifecycles.
  • Experience with automated functional GUI test tools.