AGENDA: We will focus our discussion on the following topics:

Pre Software Measurement eWorkshop Inputs

The pre-meeting questions

What experiences have you had in calculating ROI?
o At which organizational level did you approach the problem?
o On which program areas did you base the ROI calculations?
o Were your conclusions accepted? If not, what were the main inhibitors?

What are the goals of measurement in the CMM,CMMI, and Profes models?
o What is measured in these models? (consider both process and product)
o What, if anything, do you feel is missing? Why important?

ROI input materials

[C. Seaman] As I was reading through the stuff in the invitation on ROI, it occurred to me that different participants might be viewing this question quite differently. My first assumption was that we were going to be discussing the use of metrics (i.e. which metrics to use) for calculating the ROI of introducing some new technology (e.g. inspections or pair programming) into a software development organization or project. But as I read the entire list of questions, it occurred to me that some may assume we're talking about calculating the ROI of establishing a metrics program in a software development organization. These are two very different questions, and there might very well be other ways to interpret this topic. So I guess my caveat would be to watch the discussion to recognize if different people are really talking about different questions altogether. Some possible discussion questions that might help clarify the scope and get everyone on the same page might be:

- What types of decisions are supported by a ROI analysis?
- What questions are answered by a ROI analysis?
- What metrics data should be used for the cost part of the ROI equation?
- What metrics data should be used for the benefits part of the ROI equation?

[J. Münch] At IESE, as an institute for applied research we evaluate innovative techniques, tools, and methods, and package them for industrial application. In order to convince companies to introduce innovative approaches, we aim at providing empirical knowledge about the effects of these approaches in different application contexts. Being able to present proven cost/benefit data appears to be a major driver for convincing companies to introduce new approaches or change their processes. In addition, the demonstration between business goals and software approaches is important for making decisions about innovations. We also aim at observing our technology introduction projects in order to quantitatively evaluate the effects of the change and to get hints for potential further process improvements.

What experiences have you had in calculating ROI?

[J. Münch] I typically approached the ROI topic on the organizational level of departments or groups responsible for innovation and/or process improvement. We have, for instance, experience with ROI calculations in the following program areas:

• ROI of process management and improvement: In this field, a lot of quantitative material is available providing global data about the benefits/ROI of specific improvement approaches (such as CMM). Usually, this data is coupled with a large set of interacting improvement measures so that it is not clear which measures contributed to the success. It is typically is not analyzed whether there exist success factors that were either not considered or that are to be found outside the improvement program. We aim at defining ROI models for concrete technologies (i.e., techniques, methods, tools). We defined, for instance, a ROI model for process documentation that includes factors such as initial documentation cost, reduction of the annual documentation maintenance cost, and side benefits such as increased documentation quality or free effort for other activities.
• Defect reduction: We use quality models characterizing defect distributions (i.e., characterizing their origin, their removal effort, their detection time, etc.) as a means for helping companies adjust their quality strategies. Typical questions arising in industry are: Where and how much should I inspect? The removal of which defect classes is cost effective? Having an idea about the effects / ROI of changing a quality strategy helps companies to focus their efforts in this area. In addition, the problems become explicit and a lot of improvement proposals arise during the development of such models.
• Development of a measurement system for productivity: The topics of subcontracting and offshoring are currently motivating companies to define and benchmark organization’s productivity. Typically, the ROI of subcontracting tasks or developing software offshore are not clear. Applying simple productivity models considering only wages as cost seem to be not fair in these contexts because there are a lot of additional cost drivers (such as additional quality assurance cost, communication cost, etc.). A goal-oriented derivation of specific ROI models for different goals and contexts seems to be appropriate.
• Cost estimation: In my experience, calculating the cost of cost estimation is not difficult. Companies tend to minimize necessary effort (especially interview time of experts). It is much more difficult to determine the benefits of cost estimation because a comparable situation is usually missing.

My overall experience is that empirically based ROI models are very helpful for introducing software engineering technologies in practice. The models need to be focused on the technology and evaluated in the context in order to be applicable. However, technology providers nowadays provide ROI models only very seldom.

What experiences have you had in calculating ROI?

[C. Seaman] My experience with ROI has been as an instructor in software engineering and systems analysis, where I teach my students to calculate ROI in order to help their customers evaluate and compare different IT solutions. So I focus on drawing the measures used to calculate ROI from their problem statement. So if the problem that drives the improvement project has to do with reducing the effort required to perform some work process, then measures of effort should be the primary components of the ROI calculation, at least in calculating the benefits part of the equation. I think the costs part of the equation requires some brainstorming and creativity in finding relevant costs that can be measured with some reliability. This is difficult.

What experiences have you had in calculating ROI?

[O. Laitenberger] Demonstrating the cost-effectiveness of software inspection and testing technologies in different environments by different kinds of empirical approaches. The demonstration was mainly driven by scientific questions. The examinations were primarily conducted on a project level and often involved the introduction of a new technology. The conclusions were mainly accepted (scientifically in terms of published papers).
Demonstrating the effectiveness of process improvement in/reorganization of a large development organization. This effort took about 1.5 years. The demonstration was based on collected measurement data and addressed the whole organization. Top-Management was the target audience. The conclusions were accepted.

Hypotheses:

For basic software engineering technologies, such as software inspection or testing, we don’t need more ROI-studies. Despite the existing (empirical) evidence the basic technologies are often not used in industry. Hence, other factors must be relevant for their adoption. These need to be studied.
The purpose of ROI-studies from a Software Engineering engineering research perspective is to acquire more funding from either funding organizations or companies. However, few of these studies motivate the adoption of those technologies on a larger scale.
There is a general lack of practical strategies, models and measures (“off the shelf) to demonstrate the effects of process improvement in software development for decision-makers. Part of the problem is that most ROI-studies focus on the technical level and fail to show the business impact.

Process Model inputs
What are the goals of measurement in the CMM,CMMI, and Profes models?

[J. Münch] The focus of CMM measurement is on status metrics for activities. CMMI allows for measuring selected process areas depending on the information needs of an organization. The measurement purposes are manifold (e.g., “providing insight into the performance of the process, its work products, and its services”). Continuous improvement approaches based on the Quality Improvement Paradigm (QIP) such as Profes tend to be more focused to the central problems of an organization. They provide guidance on how software projects should be performed based on reusing existing experience. They are flexible with respect to process and product metrics. Profes supports the modeling, analysis, and organization of quantitative and qualitative experience for future use.

My main experience is in applying continuous improvement approaches. Below I have listed some conclusions drawn from my experience in the context of software process improvement in industry:

• Quantitative measurement should be accompanied by collecting qualitative experience (e.g., lessons learned). This can be helpful for getting valuable information that was not anticipated and helps to interpret quantitative data.
• Companies often collect a predefined set of standard measures periodically. When deriving metrics that support an improvement program, it should be carefully observed how these metrics could support the measurement goals. Introducing similar and/or redundant metrics can lead to a decrease in acceptance of the measurement efforts.
Collecting process measures requires adequately defined explicit processes that map the real processes of an organization. It is advantageous to combine descriptive process modeling with measurement in the first iterations of continuous measurement programs. Later on, process adherence should be checked periodically.

What are the goals of measurement in the CMM,CMMI, and Profes models?

[C. Seaman] My understanding of CMM/I is that measurement is used as part of the infrastructure that a high-maturity organization needs in order to achieve and maintain an engineering approach to software development. Thus, measurement is not in the same class as other software development techniques, such as inspections and reviews. Measurement in basic areas such as cost, quality, and schedule are mandated, but the real sign of maturity is an organization's ability to identify the metrics most appropriate to a particular situation, and also make use of measurement data at an organizational level.

[O. Laitenberger] There can only be a single goal of measurement in CMM, CMMI (I don’t known the Profes model – known in industry? What about SPICE?):
To show that any improvement on the maturity scale results in higher development productivity and/or product quality.
From a business perspective, this directly translates to either the development of more functionality for customers with the current number of developers or maintaining the same level of functionality with less personnel. I doubt that many are aware of this goal and the associated consequences. However, after managing some rounds of developer lay-offs due to process/maturity improvement the effects are quite transparent to me.

There is a lack of practical strategies, models and measures (“off-the shelf”) to demonstrate the effects on a larger scale and for a business perspective.

	FC-MD is a Research Affilliate of the University of Maryland.
	Please visit the Center for Empirically Based Software Engineering (CeBASE) website