Agile Software Development: An Industrial Case Study

Frank Maurer & Sebastien Martel
University of Calgary
Department of Computer Science
2500 University Dr NW
Calgary, Alberta, T2M 1N4
Canada
Phone: +1 (403) 220 3531
Fax: +1 (403) 284 4707
{maurer, smartel}@cpsc.ucalgary.ca

Based on industrial data from one company (Bitonic Solutions Inc. http://www.bitonic.ca/frontpage/index.html), we compare the productivity of agile methods with more conventional approaches. Bitonic was using ad-hoc OO software development practices (use cases, UML design, etc.) before adopting agile techniques, mostly XP techniques, last year. Besides switching to lightweight approaches in 2001, the company context stayed more or less stable over the timeframe providing the data for our investigation: the data was gathered from work for the same client on the same product and the team was stable over the time when agile approaches were introduced.

Bitonic deviates from the XP process described by Beck [1] in the following ways:

• Customer Involvement: Two key employees of the client's company represent the customer. The customer representatives are not co-located with the development team but are in close proximity in the city so that they are accessible fast when telecommunication is not sufficient to resolve issues. 
• Estimation and Metrics: In the course of the current iteration, user stories are created and sent to the project manager to be estimated. They are then placed in a specific iteration immediately or during the next meeting with the customer representatives. The project manager asks a programmer to provide an initial estimate. The effort estimation is put in Bugzilla as a comment. Bitonic uses Bugzilla as the primary means for project coordination: they track bugs as well as new feature requests (new user stories) with it. After gathering the estimates, user stories are brought to the next meeting to be discussed with the client. This process creates an extensive amount of communication between the project manager, the client, and a programmer just to estimate a task.
• Documentation: Even though Bitonic has drastically reduced the amount of analysis and design documentation since adopting XP, they recognize the need for a certain amount of documentation. The documentation is used to get approval from the client executive committee for implementing new features and major tasks (e.g. major redesigns etc.). Bitonic estimates that for 12hours of programming they spent about 1hour on requirements/design documentation (using text and UML).
• Testing: Bitonic is using JUnit to aid in their unit testing task. Sometimes, tests are still done after the production code because time to the next release was approaching fast and the team wanted to ensure that the feature will be present in the next release. Additional quality assurance is provided by the customer company on the same level as it was before Bitonic adopted XP practices.
• Pair Programming: Pair programming is only done for major task. Bitonic estimates that 50% of the production code is written using pair programming. The team members that used pair programming liked it.  The lower level of pair programming (compared with standard XP practices) has caused the creation of programmers that are very knowledgeable in specific parts of the system.

EMPIRICAL RESULTS

In this section, we summarize the empirical findings of the case study. A larger paper [4] that describes more details (including the exact definition of the metrics) is available upon request. Size metrics were gathered from the configuration management system for 16 monthly releases. 5 releases were developed using agile practices, 9 releases followed their original process, 2 releases were in the transition phase and were excluded when we computed the average results below. As effort we used the number of hours billed to the customer.

Table 1 Productivity averages

Overall, if productivity is defined on hard size metrics (LOC, #methods, #classes), the results indicate strong productivity increases after adopting agile practices. If productivity is defined based on number of features delivered to the customer, productivity decreases are visible after adopting XP. The later results are unexpected and disappointing for agile approaches. Our paper [4] discusses this in more detail. The summary of the discussion is that Bitonic is now addressing much more complicated features than in the beginning of the project and, hence, would argue that a 4.96% decrease in the number of features delivered to the customer per hour in fact is an increase of their development productivity.

REFERENCES

[1] Kent Beck. Extreme Programming Explained: Embrace Change. Reading, Massachusetts: Addison Wesley Longman, Inc., 2000.

[2] Kent.Beck, Martin Fowler. Planning Extreme Programming. Reading, Massachusetts: Addison Wesley Longman, Inc., 2001.

[3] Ron Jeffries, Ann Anderson, Chet Hendrickson. Extreme Programming Installed. Reading, Massachusetts: Addison Wesley Longman, Inc., 2001.

[4] Frank Maurer, Sebastien Martel:  On the Productivity of Extreme Programming: An Industrial Case Study, http://sern.ucalgary.ca/~milos/Library.htm

[5] James Newkirk, Robert C. Martin: Extreme Programming in Practice, Addison-Wesley, 2001.