Summary of the First eWorkshop on
Agile Methods
April 8, 2002
Part 4
Experimentation
People seem to believe that it is obvious that agile
methods work. Marv Zelkowitz asked if there was a need for validation. Despite the perception that agile methods are obviously
beneficial, some participants (Philip Johnson and Alistair Cockburn) felt that
there is a need for proper validation (especially for late adopters) and
suggested possible studies.
Is experimentation important?
Some participants (6) agreed that experimentation is
important. Laurie Williams pointed out that her experiments results
have helped to overcome resistance to pair programming. Don Wells said he
experienced the same thing; “We
had to get PP going by mixing people up till something mysterious and magical
happened.” Ken Auer mentioned that experimentation is extremely useful to help
people get over their fears... it's OK to stumble in an experiment!
Scott Ambler believes that experimentation is important for many in the "late majority" and "laggards" on Moore's adoption curve. "If it ain't proved I ain't interested" is a common refrain among these folks. People at this workshop are early adopters or innovators and are willing to try new things. But, there are many organizations that are still struggling with adopting OO, let alone agile stuff. Many participants (12) agreed that the late adopters would need to see data before they would use agile methods. Peter Hantos believes tha since the application of these methods is so politicized, controlled experiments, and particularly if they are conducted in a university setting will miss those cruicial political and contextual points. Alistair Cockburn added that they may ignore the data from experiments, but if you don’t have data, they will ignore you.
Alistair Cockburn asks for proper validation. At the USC workshop, his suggestions were turned down, because they were "too obviously true". But we still need the data to show people, otherwise they won't believe.
Philip Johnson agreed but believes it will be difficult to
experiment with agile development because the method is under specified.
What kind of studies you would like to see?
The following studies were suggested:
§
Scott Ambler: Exploring the effectiveness of manual
modeling techniques (CRC cards, white board sketches, low fidelity prototypes)
The idea of discarding temporary models and traveling light should be
addressed
§
Hakan Erdogmus and Joel Martin: Experiments to measure how
information about the system, the process, and technical skills are propagated
through the team using agile practices
§
Don Wells: some sort of experiment in cognition versus
strict plans or looser collaboration styles.
§
Alistair Cockburn: I want to see quantitative studies
comparing team morale / citizenship / communication paths used, to project
outcome. My prediction is that they link better than most others (other than the
standard project killers). Without studying those factors, we only see the
"Process", and the important factors slip through the survey's net. (Several
participants were interested in this experiment – Scott Ambler, Winsor Brown,
Tim Mackinnon, Ken Auer)
§
Bil Kleb: BDUF vs Test Driven Development designs
§
Alistair Cockburn: Study of agile and architecture
evolution
Some of the participants felt that it might be difficult to
experiment with agile methods using controlled studies and suggested case
studies or less formal experimentation:
§
Forrest Shull said: If
controlled experiments are hard/unfeasible to do right now, we could get a lot
of value out of case studies. It would be great to say that we followed an agile
methodology group and document that it achieved a highly dependable real-time
system (or whatever it is that objectors argue against agile ever being able to
reach. Peter Hantos and Barry Boehm agreed.
§
Hakan
Erdogmus and Joel Martin stated that not
all empirical studies must be "controlled". A great deal can be learned by
careful, qualitative observational studies by people who know what they are
doing. The "controlled" bit is necessary for making generalizations about
causation. That is not always possible.
§
Scott
Ambler agreed and added: We need to keep things honest. However, someone who
knows what they're looking at needs to write up the failure. In the mid-90s
there was a lot of articles about how you couldn't store objects in relational
databases, but they
often blamed lack of experience with C++ on mapping objects to RDBs. I'd
hate to see teams blaming a process for their own mistakes. Software development is fragile,
one bad decision can kill your project. A common bad decision is to go
against the iron triangle and try to set the level of quality, the
level of resources, and the scope of your effort -- at least one of these
factors has to vary.
Documentation
Is any documentation necessary at all? If so, how do you
know how much?
There was also some discussion about the use of
documentation with agile projects. Scott Ambler commented that documentation
becomes out of date and should be updated only “when it hurts”. Documentation is
a poor form of communication, but sometimes you need it to retain critical
information over time. Many organizations demand more than is needed. The goal
should be to communicate effectively and documentation should be one of the last
options to fulfill that goal. Barry Boehm mentioned that a documented project
makes it easier for an outside expert to diagnose problems. Kent Beck disagreed
saying that as an outside expert who spends a large percentage of his time
diagnosing projects, he is looking for people “stuff” (like quiet asides) and
not technical details.
Bil Kleb said that with agile, documentation is assigned a
cost and its extent is determined by the customer (excepting internal
documentation).
Suggested reference: Scott Ambler suggested his Agile Documentation essay as good reference for this topic.
Next
eWorkshop
In the end of the meeting, the participants were asked
whether they were interested in another workshop sometime in June. Most of them
had a positive response.
[Main Page] [Part 1] [Part 2] [Part 3]