Summary of the Second eWorkshop on Agile Methods

The first topic of the eWorkshop was about introducing Agile Methods in organizations. The participants discussed their experiences and gave some suggestions on how to handle specific situations.

Ken Schwaber mentioned that one situation when introducing Agile doesn’t work very well is in organizations that are "just looking". For those, generally Agile does not work so well. "Just looking,” means that they are interested, but their back is not against the wall because of a visible project failure (IT organizations) or profitability (product organizations).

Scott Ambler said that rigid thought processes in general are a problem. He also mentioned that he is currently trying to introduce agile techniques into the “data community” and he is experiencing pushback by some people that don't want to consider new approaches. (He suggested http://www.agiledata.org) For those people, BDUF works fine for them. They're also in positions of power, and agile threatens that. Narti Kitiyakara stated that they have had problems with getting both process-oriented and non-process-oriented people to accept Agile Methods. Roseanne Tesoriero mentioned that she has had some difficulties with introducing XP to undergraduate students.

Tuomo Kahkonen mentioned that introducing agile methodologies to teams that are a part of a large organization has been difficult. When there are many-to-many relationships between customers and development teams, it is hard to establish the “customer-always-available” practice.

Marvin Zelkowitz pointed out that establishments resist any change. Gary Pollice said that any change is difficult and it takes some help to implement it. That is the biggest thing he has found for change in general -- get help (a mentor or coach) to help ease the transition.

Hakan Erdogmus described a positive experience in a research organization, but warned about the organization’s questions on how to adapt the new method to their environment. The introduction was bottom-up and they faced no resistance from management. The new method was well received for both technical and team practices; but not so well received when it came to actual and full-blown use in projects. “People want to introduce XP, but are not quite sure how to adapt it to their own environment.”

Ken Schwaber said that he has had successful introductions in situations where management introduces Agile Methods and the project is critical to the organization.

Tuomo Kahkonen said that in order to introduce Agile successfully, he has been using a pattern approach to describe the problems the practices solve and the related practices. He has classified XP and other practices related to people, development and collaboration.

Scott Ambler believed that the “data community” would often admit to the fact that people will pay lip service to their “enterprise models”, their standards, and then go off and do their own thing anyway. In order to avoid unsuccessful introduction of Agile methods to them, he is pushing the idea that handing off models and documents is not the answer. Instead they need to actively work with the development teams in an iterative and incremental manner.

Bil Kleb suggested the Satir model of change (http://www.stevenmsmith.com/articles/satir_change_model.htm), which includes an anticipated phase of "chaos", as an important concept to introduce early in a transition. He also mentioned a group of “process-burned” folks, so to introduce Agile, talking about things from a process point of view was something to avoid.

Following Ken Schwaber’s comment about “just looking” organizations, Gary Pollice concluded that the best time to introduce a shift to a more agile method is when the project has it's back against the wall. Ken Schwaber agreed because then management is willing to accept the change required.

Gary Pollice asked participants who they’ve gotten to accept Agile Methods so far. This lead the discussion to the introduction of Agile to management.

Apparently depending on how Agile is introduced, management and developers have different reactions. Narti Kitiyakara said that developers have appreciated the practices, but managers seem more reluctant. One of their managers is very process-oriented and tends to view XP as a simple thing; the other is not process-oriented and found it burdensome. James Grenning has seen that if agile (XP) comes from the developers, then management is suspicious. If agile comes from management, then developers are suspicious. Narti Kitiyakara didn’t believe that was the case for them. In their case, XP was introduced by management with a mix of reactions on both ends.

Scott Ambler said that he got an email a few months ago describing a project team that tried to get XP introduced, but management didn't want to hear about it. They explored the issues and then discovered that modeling and documentation was a serious consideration for management, so they tailored Agile Modeling (AM) (http://www.agilemodeling.com) into their proposal and got it through.

James Grenning stated that a willingness to try new things helps with success. A rigid management makes it very difficult for the teams to make progress at managing themselves. Ken Schwaber reminded participants that at XP2002, Kent Beck called this the year of the manager because we are going to try to get management to start accepting agile as a regular practice for normal development.

Barry Boehm brought up the topic of incremental introduction in reference to rigid thought processes. He said that some agile method leaders are rigid in requiring full adoption. He also added that Jim Highsmith says that a big advantage of agility is the ability to apply any of the methods selectively and generatively. He asked the participants if they agree or disagree and asked for comments on experience with partial adoption.

For some participants, partial introduction/adoption works or not depending of the agile method. Ken Schwaber pointed out that Agile such as Scrum and XP are interlocking, co-dependent processes. Scott Ambler mentioned that many XP folks are pretty adamant about you adopting the entire kit & caboodle. With Agile Modeling, he distinguishes between core principles & practices and supplementary ones to make it easier for people to do partial adoption. In Ken Schwaber’s opinion partial adoption works in the methods that Jim and Alistair describe (such as open space and communications factors). But, when Kent, he and others built XP and Scrum, they put the practices together to mutually support each other. They aren't plug and play because then you lose the control and effectiveness built in. A common XP problem is that people don't implement refactoring or pair programming. Without these, the rest starts to put out lower quality code. James Grenning mentioned that he has seen that when refactoring is neglected, it does not take long for the team to feel the pain. Once they feel the pain for themselves, they take it more seriously. Also a general lack of design skills makes refactoring difficult. If the team does not recognize good and bad design, refactoring does not get you anywhere.

Some participants believe that partial adoption is effective or at least better than nothing. Bill Wood said that he has seen an improvement by introducing practices one at a time. Yet, he believes full implementation is certainly best. Gary Pollice agrees that with agile methods you have the ability to apply any of the practices selectively and generatively. Peter Hantos believes that applying methods selectively is a tactical advantage. On the other hand, this selectivity causes the failure of many CMM-based improvement efforts. The notion of "following the spirit" is in most cases only a cop-out. Gary Pollice believes that incremental adoption of best practices works best. When people try to adopt things like RUP incrementally they seem to have good success. Tuomo Kahkonen mentioned that in many cases, XP as such is not a feasible alternative, but teams can increase its agility by introducing some of the practices. He believes partial introduction of practices is better than nothing.

Some participants believed it is feasible to do a partial introduction as long as people know all practices before start dropping them. Bil Kleb belives that for partial introduction to work, you need to intimately know all the techniques before you start dropping or choosing. Ken Schwaber added that when they introduce Scrum, they tell people not to become "process" experts until they have used it and understand it. Then they can start adjusting it to their needs (like 15 day Sprints instead of 30 day sprints)”. Narti Kitiyakarabelieved that you could get advantage from many of practices without being fully aware of all of them. Bil Kleb pointed out that you definitely lose synergy in that case. James Grenning said that they try to get organizations to try all of XP. That helps them find what is easy for them. At that point, they often focus on those “easy” practices and work on the tougher ones later. But he still believes that at least some of the folks should have experience with all the practices under consideration so they can recognize when one of the omitted practices would bring value.

Some participants didn’t believe it is feasible to have partial introduction. Hakan Erdogmusdoesn’t think you can simply eliminate practices at will (for example, deciding we won't do PP because people don't like it). He believed most of the benefits emerge from synergistic effects. For example, you can't give up collective ownership, pair-programming, test-first, and continuous refactoring and expect to produce quality code. Ken Schwaber said that the danger of partial implementation is that the control and risk practices of agile are absent; if not careful, this results in hacking.

Vic Basili thinks you have to be very careful what set of activities you use - many of them play off one another and so if you introduce them very carefully, you can do it incrementally, but he would prefer to do a small project fully. Ken Schwaber shared his experience changing IT organizations by implementing Scrum/XP in three or four critical projects, and then letting the rest of the organization observe. They then start picking up on it, manager by manager, project by project.

The Moderator proposed a vote questioning if Agile Methods can be introduced incrementally. The results of the 20 votes from participants showed that their opinion was divided (11 yes, 1 no and 8 not sure).

The participants discussed which practices should be introduced first because they are easier to get acceptance for.

Coding standards and code ownership where mentioned as easy to get acceptance for. In Bil Kleb’s opinion, coding standards is almost the easiest “sell” once a clear reason for the need is presented. Narti Kitiyakara mentioned that collective code ownership hasn't been a problem. They do not pair as effectively as they would like, but they don't believe it's related to the coding standards.

Pair programming and test-first were considered hard to get acceptance for. According to Hakan Erdogmus, pair programming is a hard sell until folks try it (like the famous Green Eggs and Ham book). In his experience, out of about two dozen people he knows, only one who tried pair programming didn't like it. Bil Kleb believed that test first is a hard sell until folks try it. In his opinion, refactoring, once testing is in place, just “blows folks away”.

There was some discussion regarding which practices that are essential for the process to work. Hakan Erdogmus belives that collective ownership, refactoring, test-first, automated testing, and PP is a major subset. He'd argue that this is a substitute for inspections. Jan Hunnius added that as there are a lot of one-way dependencies of practices within about three dependency-groups, he believes certain orders exist in which to successfully introduce practices of XP, even one by one (orders to be defined and to be proven to work). They thought about three groups of practices of XP: Customer related, code related, and others.

The participants then voted on a number of statements from other participants regarding what practices are related to each other. The voting results showed that there wasn’t a lot of consensus. Some participants mentioned that the statements were ambiguous or that it was hard to vote on them with a yes or no answer. Because of that, the votes are not included on this summary. However, all participants agreed that you can’t do XP without Unit Testing.

Some participants shared their experiences with integration of CMM(I) and Agile. Pete Nelson mentioned that they used XP before their CMM implementation. They were able to implement the spirit of the CMM without changing (too much) the way they were developing software. They were using XP with good success. The CMM helped introduce it. Karen Smiley said that they are trying to introduce XP while we are in the process of transitioning ABB worldwide from CMM to CMMI. They are in the opposite position from Pete - CMM[I] was introduced several years before their XP initiatives. This is true of their corporate research centers (CRCs) as well as the business units. Bill Wood said that they’ve seen a better match with CMM and agile. The CMM part is worded generally, as in "follow a practice of choice", and not delving into specifics such as, "must have spec sheet 5 pages long."

Aldo Dagnino mentioned that his position paper states that Agile Models can be compatible with CMMI. They have incorporated agile practices to an evolutionary model and they are mapping it out to CMMI and they expect that it will have compatibility.

Karen Smiley added that their organization has adopted the CMMI framework and they are incorporating Agile practices to the evolutionary development lifecycle model. If they do this, they believe that there is a clear distinction between life cycle models and continuous process improvement models such as CMMI and both are not incompatible.

There was some discussion regarding the importance of data to introduce Agile methods.

Scott Ambler said that an interesting theory that he's been pushing is the relationship between Agile Methods and Moore's tech adoption curve. A lot of people seem to be asking for proof that it works, this is true of many in the data community it seems, yet all that exists for the most part right now is anecdotal evidence. This is great for innovators and early adoptions, not so good for the late majority and laggards. For him, the data community is not too interested in new approaches and they are experiencing “pushback” from them. They might be worried about loss of power. They just do not want to consider new approaches and BDUF works fine for them. They're also in positions of power, and agile threatens that. People in positions of power are less likely to support change if it means they give up that power.

James Grenning stated that data is hard to come by. He believes it would help. At this point most of those that demand data are looking for reasons not to change. Scott Ambler agrees added that he is seeing this a lot with the "hard core folks" in the data community. They're happy with things the way they are, regardless of whether it actually works well. What he is seeing is that people want to see "proof" that agile works before they adopt it. It's the lack of data that gives them the excuse not to consider changing. Ed Colbert disagrees: “If the data shows benefits, many would jump on the agile approach.” James Grenning believes that those having successes know they are better even without a big metrics program.

- Skeleton of database of XP and Agile practices being built in the framework of the EU NAME project: http://www.case.unibz.it/XPUT. Suggested by Giancarlo Succi.

The discussion on the second topic, nonfunctional requirements[1] (NFRs), seemed to indicate that there were two main trains of thought among the participants. Unfortunately, differences between the two camps were not reconciled successfully by the end of the discussion.

The central issue was how NFRs can be effectively handled in an agile development environment. In contrast to functional requirements, which describe what a given software system should do, NFRs attempt to express something about the properties of how that behavior should be performed: for example, by mandating upper bounds on performance time, specifying safety or security constraints that must hold true, or stating certain reliability expectations for the system as a whole.

Several participants felt that agile handles these quality concerns through the emphasis on test-first development (Ambler, Erdogmus, Grenning, Kleb, Narti, Schwaber, Wood). For any nonfunctional requirement, the developer must figure out how to state an appropriate test case, which will be entered into the acceptable test suite just like any other test for functional behavior. Agile’s reliance on comparing each build of the system against the test suite means that the developer can always understand if the nonfunctional requirements are being satisfied.

Some examples from practice were given. Bill Wood and James Grenning stated that they have found that test cases can be written to cover NFRs concerning issues such as the number of users and limits on memory/cpu usage.

Bil Kleb proposed a secondary benefit of agile planning processes for handling NFRs: by turning NFRs into user stories, they can be subjected to a cost/benefit analysis just like any other requirement, helping developers to get a handle on how important they are in relation to the likely effort to be spent implementing them.

However, not all participants were satisfied with this approach. Gary Pollice stated that he did not believe that NFRs can always be restated in such a way that they can be covered by a certain number of discrete entries in the test suite. In particular, he highlighted performance requirements as difficult to handle in this manner. Barry Boehm seconded this remark and mentioned that he felt strong examples were requirements dealing with throughput and response time. Because these are difficult requirements to ensure in any case (since measures may have to be taken over long time periods, can depend heavily on hard-to-predict interactions with other modules or system load, etc.), he did not think that simply making additions to the test suite would be sufficient to address these concerns.

In the end, a vote showed that participants were not strongly certain that all NFRs could be covered by test cases. (1 participant felt they could always be expressed as test cases, 8 disagreed, 4 were not sure.)

The only approach other than writing test cases to be mentioned came from Ken Schwaber. He has done several projects with Scrum that had NFRs in the areas of scalability and security. These were stated in the product backlog and kept as top priority items that had to be considered when writing any function.

A strongly related point had to do with the feasibility of achieving certain NFRs without investing in Big Design Up Front (BDUF). Barry Boehm felt that the main difference between NFRs and functional requirements is that NFRs may in fact require early architectural commitments. He used the example of requirements relating to security: If developers following the principle of You Aren’t Gonna Need It (YAGNI) create a first increment of a system with as simple a design as possible, it is highly likely they would have to throw away the whole design in later increments when they need to implement the security constraints. Security constraints require a lot of up-front commitment in the architecture and are not easy to simply add-in to later versions of a system. A second example was performance scalability: If developers start simple with a non-scalable 4GL, they would have to throw it away to get scalability into later versions.

A dissenting voice came from Scott Ambler, who suggested that such quality constraints could be added to a system in later increments through refactoring the architecture, rather than throwing away the existing one and recreating it.

During the discussion of topic 3 the issue of GUI testing in Agile Methods was raised and there was some discussion of whether GUI testing in Agile development is different than GUI Testing in Traditional Development Methods. Scott Ambler believes there is no difference. The development approach regarding what to put in the GUI might change. In Agile Development you are supposed to keep the GUI thin and this a good practice even in non-agile development. Bil Kleb also believes GUI testing is no different in Agile Development, as long as you keep GUI code thin. Ron Jeffries believes that almost everything in Agile Methods is just like a technique found elsewhere, therefore there is no difference. Peter Hantos agreed. Ron Jeffries added that Agile is more about the frequency of the feedback than the type. Scott Ambler agreed and added that communication is also important in Agile. James Grenning also believes that GUI Testing in Agile Methods is that same as doing it in some other process(es). Peter Nelson believes that the processes are very similar but more frequent in Agile.

The moderator proposed a vote to the statement: “GUI Testing in Agile Methods is exactly the same as doing it in any other process”. From the 16 participants that voted, 6 said yes, 5 said no and 5 said not sure. There was some discussion regarding the statement. Scott Ambler suggested that other processes might not include testing at all and therefore the statement was dangerous. Many participants agreed. Some participants didn’t agree with the word “exactly” in the statement. Based on the discussion of what the statement should have been, it seems that the majority of the participants believes that best practices from other processes can be used in Agile and there is no need for an Agile Methods specific test-method. Based on the participants’ feedback, the moderator proposed another vote: “Are different methods used in testing GUIs in agile than in other environments?” From the 14 participants that voted, 1 said yes, 5 said no and 8 said they are not sure. Bil Kleb said that testing the GUI itself is no different, but what's in the GUI probably is a lot different. In response to the question Ron Jeffries believes that different methods are used in Agile, since everyone does something different, but different techniques are not needed since good techniques are good. Agile would favor the more automated, and the more flexible.

Atif Memon raised the question on how to do test-before-coding for GUIs. Narti mentioned that for HTML testing, they make an XML mock-up of the page that is transformed into HTML. This can then be compared with the HTML actually produced. He doesn’t see why you couldn't do something similar for a standard GUI, although it might require more work. To test GUI events interactions before coding, Narti mentioned that they’ve been using the java.awt.Robot. Hakan said that the test code is a use case written in the programming language itself. So you blend writing tests with micro-design in test-first.

Karen Smiley asked if there were any special considerations for applying test-first development to GUI development. Ron Jeffries replied saying that there may be some special techniques, such as the one mentioned by Narti, screen scraping, etc. He added saying that he likes to keep all functionality out of the GUI and test "behind" it as much as possible. Scott Ambler agreed and added that you need to make sure the UI still conforms to what your users want to see, to standards, etc. You also need to ensure that UI issues don’t get accidentally dropped or added for that matter. In short: you still need to test. Narti added that they like to keep the GUI thin too (it's HTML, after all), but their customer finds a lot of value in detailing exactly what a page should look like.
James Grenning said that we should make the UI thin, have only presentation logic in the UI and put all the application logic in the application and this relates to any UI development. Bil Kleb agreed.

Another topic of discussion was regarding the maintenance of acceptance tests. In answering the question: How do you modify the acceptance test suite to stay current, Bill Wood replied that we either write more acceptance tests or refactor the tests (before changing code, of course). Scott Ambler agreed and Hakan said you refactor acceptance tests just as you do your code

The Moderator asked if there was a way to do regression testing of GUIs quickly in an environment where you do daily builds. Bil Kleb suggested making the GUI so thin that testing is not necessary. Peter Hantos raised the point the automating testing is not specific to Agile domains and in all software development approaches the rule is if you can automate it, do it. Gary Pollice, Scott Ambler and Ron Jeffries agreed.

Vic Basili asked if there were any guidelines for optimizing the acceptance tests if they are expensive to run. Participants seemed to say it is expensive and you should just pay the costs. Bill Wood mentioned that they have several suites that run continuously, some taking a short time to test simple paths, and others taking hours for full stress tests. Bil Kleb said that we shouldn’t skimp on computational power, and plan to devote some resources to just sit there and check-out, build, test, check-out, build, test, etc.

Vic Basili raised the issue of finding a real customer to run acceptance tests and provide feedback. “Can we characterize projects for which it is hard get a customer to cooperate? Can we always substitute someone playing the customer role? What are the strengths and weaknesses of such a substitution activity?”

Bill Wood said you really need the customer; without the real customer on hand, you do a lot more over-development, guessing about what might be needed. Ron Jeffries believes that a substitute customer may not know the domain so well, may not have the support of the real users, may not have authority.

Peter Hantos mentioned that not having a dedicated customer is a given in the commercial (market-driven) domain. Finding willing and appropriate customers for beta tests is always a challenge. Product development is driven by using both customer/end-user feedback and market and technology projections. Narti agreed and mentioned a project destined for shrink-wrapped release where they are having trouble getting a real customer. The requirements are defined by technical personnel with no obvious feedback from either the sales department or real users. Ron Jeffries and Scott Ambler disagree that not having a customer in the commercial domain is a given. Ron Jeffries knows about a commercial product development with dedicated customers. Scott Ambler has a client that brings their customers on site, and they're doing commercial software. “This is just a common excuse in the commercial software world”.

Karen Smiley mentioned that in many cases, the customers are geographically remote from the developers. Also, in many cases pilots can be difficult to arrange with software for any activity that's mission critical. Scott Ambler suggested a reference: http://www.fastnloose.com/cgi-bin/wiki.pl/dad.

Barry Boehm said that in general, especially with agile but also with plan-driven, things go better when the customers are knowledgeable, empowered, representative, collaborative, and committed. Ron Jeffries and Bil Kleb agreed. Barry added that a good example of a well-managed XP deviation is in the ICSE2002 paper, "Recognizing and Responding to "Bad Smells" in XP”.

Karen Smiley mentioned that if you can't get the customer consistently involved, it's almost more important to have real customers available during requirements development and elicitation. It is better to get a good understanding up front of what they really need, than to try to catch it at the end during acceptance tests (although they're still needed). This is something to consider if customer availability to contribute to a project is limited. Ron Jeffries disagreed because this assumes requirements don't change and that real customers know what they need and like.

The moderator proposed a voting: “If you don't have a real customer Agile Methods loses a great deal of it's value” From the 13 participants that voted, 8 said yes, 3 said no and 2 said they are not sure”. Ron Jeffries suggested that real should be changed to good. He believes that a good business case, though not a "real" customer, could be a great customer, for example

Resources

Paper that describes a large software development project that used a modified XP approach (suggested by Barry Boehm).

Paper about dispersed Agile Development (suggested by Scott Ambler) http://www.fastnloose.com/cgi-bin/wiki.pl/dad.

[1] Some participants preferred to use a different terminology for this same concept. For example, Scott Ambler preferred the term “technical requirements” since “nonfunctional” can mean “not working.” However, in this summary we have preferred to use the more common term “nonfunctional requirements” for clarity.