Handicap Solution

The Challenge & History Handicap Components

The Solution

Not finding anything off the shelf I began looking for common analogs to pull from. Early on I grabbed hold of the concept of a handicap from Golf. A handicap is a measurement against a standard, in the case of a golf player how far they are from playing "par" golf on average. This allows two golfers to able to compare the success of their games to one another even when they have two radically different handicaps. Using the handicap they can reduce their scores to a common evalation point to see if they are playing a better, worse or average game for the day.

The output of performance tests is always a function of the inputs. These inputs are both on process and a mechanical basis. For instance, if management refuses to sign off on the load profile then there is a pretty good chance that someone believes the model of load is wrong or that management is not supportive of the efforts. If you don't have enough time to properly construct the test, then shortcuts are invariably made to meet a deadline. There is a common phrase for this in IT, "garbage in...garbage out." This increased risk in the effort should be captured as a measured element.

Lastly, we need a number, a score, a measurement against a known standard, to allow for management to understand how far off the standard the current test in play falls and therefor how much risk is involved in deployment. I decided on score range of 1-1000, with two measures, one for the planning and another for delivery with the actual tests. I termed this my test confidence index. Others have called it a risk score. The higher the value the better There would be some elements common to both measures, and one section distinct to planning.

While the range is 1-1000, a perfect score is no considered achievable simply because of the compromises made in test to hit the delivery targets. Rules of thumb:

  • The higher the score the better
  • Valid tests target a range of 750 or higher
  • Tests below 400 indicate severe compromises in test quality
  • The Planning Handicap

    The planning handicap consists of four sections, each with a total score of 250 points:

  • Environment
  • Data Model
  • Load Model
  • Planning & Scheduling
  • The Test Handicap

    The Test Handicap drops the Planning & Scheduling section. In order to obtain a normalized score to the 1000 point scale the sum of scrores for the first three sections is divided into 750 and multiplied by 1000

    (Environment Score + Data Model Score + Load Model Score) / 750 * 1000

    Alex Podelko has suggested that the assumption that the test is sound in construction may be a dangerous one. As a result of that observation I have added an alternate section entitled Podelko's Suggestion as a check against poor practices in the test implementation where other sections may near perfection in implementation

    Derivative Works

    Please take this body of work as inspiration for implementing your own measures on internal quality. I fully expect readers of this document to modify, extend, contract this body of work as needed to report on test quality within their own organizations. A key attribute I would recommend you retain is a grounding in objective data where multiple people can run your model and arrive at the same result. A subjective model will become the subject of debate on the relative values of the scores for configuration of OS and services. Non IT management types also tend to prefer reducing complexity to a number and a scale to measure success by.

    The Challenge & History Handicap Components