Statement 3: Agile methods fit applications that can be built quickly and don't require extensive quality assurance. Agile methods work less well for critical, reliable, and safe systems.
a. Do you have experiences or data that confirm or refute this statement? (Briefly explain)
Confirms - Agile methods do not facilitate quality assurance.
Peter Hantos:
The need for QA relates to scale because when a complex system is integrated, thorough unit testing of the components does not guarantee the achievement of system-level quality, reliability or safety requirements.
Gary Pollice:
My experiences are that some techniques, that are considered agile might be applied in almost any situation. One example is the test-first practice of XP. This can be used in any development situation I’ve encountered. However, when there are high criticality, reliability, and safety issues, more than this is needed.
Refutes - Evidence for use in systems that can't be built quickly.
Tim Mackinnon:
XP has proved extremely good for an application that we thought we could build quickly but which actually we are still building 3 years later. While not safety critical we spend a lot of time doing extensive testing (automating as much of this as possible).
Refutes - Evidence for use in business-critical systems.
Frank Maurer:
We do not have any experiences related to safety critical systems. The system that our industry partner was developing was business-critical for their customer. Their customer (an e-business company) is still in business. Hence, I believe that agile methods are useful for building reliable systems as long as (a) the system has a user interface (b) requirements change quite often.
Refutes - Evidence for use in extensive quality assurance environments.
Ken Schwaber:
Numerous high quality systems have been built using agile processes. My most recent specific experience was building a FDA qualified teleradiology system that delivers life-critical information in medical care settings.
Refutes - Agile methods work fine for systems requiring quality assurance.
Scott Ambler:
Seems like you've connected two orthogonal issues - QA and safety.
Why do you think that QA and agile do not go hand in hand? For example, XP takes a different approach to QA but still performs it - a side effect of its approach to pair programming is that many people see the source code, alleviating the need for walkthroughs and reviews. I'd argue that the code has still been QA'd, just in a different manner.
Agile Modeling tells you to Maximize Stakeholder Investment. If QA activities are a good investment, and done right they are, then do them. Just recognize that there are costs and benefits.
Erdogmus & Martin:
Joel's experience with JUnit suggests that test-first techniques may produce superior-quality tests for small systems than plan-oriented methods can. When the emphasis is on planning rather than test-first, programmers/customers don't tend to review, improve, or refactor test code that they produce. For three small systems, the emphasis on refactoring test code led to important corrections in the test code. This effect, if valid and scales up, would be an advantage for applications with stringent quality requirements.
Randy Miller:
Agile methods can be used to build a variety of application types. Certainly agile processes are faster at delivering applications (hence the name). But quality should not be sacrificed. Quality is a function of many variables. Some of these variables have to do with process. However, most of the better-known (and well-defined) agile processes cover these quality basics. If your system requires mission critical quality, you may want to augment agile processes with additional risk mitigating activities. Any process would require this. By the way, delivering space shuttle software is a different animal than your typical Java app and also requires different skills and experience.
Don Wells:
The premise has not been proven. Extreme programming encourages QA to be an integrated part of the process and does not prohibit further QA in a conventional throw it over the wall style.
Bill Wood:
I have only worked XP on short-duration projects, but have experience on projects lasting more than a decade. This software is used to certify human-rated launch services, such as the space shuttle. Inaccurate data from these codes can lead to the loss of $50M of hardware for a Mars probe to $5B for the space shuttle. Agile methods emphasis on tests, integration, and flexibility are benefits for these types of critical, reliable, and safe systems.
b. Can you refine the statement based on your experience (e.g., works well for reliable systems but not for secure/safe ones)?
Scott Ambler:
This depends on the method and the team. I think the issues of agility and safety are orthogonal. In Barry's article I was quoted out of context, likely my own fault, discussing safety. What I meant was that I don't work on life-critical systems (by choice) and as a result have not used Agile Modeling in such environments. Nor do I know anyone who has yet. Therefore, I am not in a position to recommend that Agile Modeling be applied in this situation.
Peter Hantos:
Leave out the second sentence. Agile Methods do not work at all in those situations, because some fundamental concepts clash with the necessary detail and rigor needed to satisfy the higher-level criteria.
Don Wells:
Agile methods produce systems that have been well tested at the very least. I am unfamiliar with methods that go beyond this.
c. Can you cite any publications that directly address this statement? (Give citations)
Tim Mackinnon:
Computer Weekly (UK publication) has over the last 5 years published many articles on the problems with the NATS (National Air Traffic control System) - this system was considered safety critical, went hugely over budget and had enormous bug rates. Just last week the system was switched to manual due to some software malfunction. This project didn't use any agile methodologies and supposedly used Plan Driven Methods, which don't look to have helped in any way.
d. What are the implications if this statement is true? How should we change development practices to address this problem?
Scott Ambler:
Pick the right process for the job.
Erdogmus & Martin:
· Intermediate releases no longer generate value if it is not possible to incrementally field the system. If the system is cannot be fielded unless it's complete and has all the critical functionality, intermediate releases serve merely as proof of progress towards a complete solution. Then early delivery with variable scoping no longer override complete functionality.
· Greater onus on customer being able to produce high-quality, comprehensive acceptance tests and to prioritize features. This may imply serious analysis and planning effort at least by the customer. Otherwise, agile methods may miss infrequently-used, but critical features due to their tendency to put timely delivery before full scope.
· The solution space for real-time, safety-critical applications often is not large, and cost of change may be prohibitively high in the case of an unviable solution. Upfront design/planning/exploration may be necessary to resolve technical uncertainty early and steer the system in the right direction.
· Software built by agile methods may be extremely reliable most of the time, but in the case of safety- or mission-critical applications, they must also ensure coverage of highly improbable behavior resulting from complex interactions among system components. How much test-first can help? Must this be done before building the system? Can it be done as a slow, methodical reevaluation of the existing system, adding oversight through refactoring?
Peter Hantos:
As I mentioned earlier, small projects should have the freedom to chose whatever methodology, tools and processes they find fit for the task on hand. On a small scale it is possible to keep the above-mentioned issues visible in form of quantified, non-functional requirements, because the total number of requirements - even adding these - is still fairly manageable. Nevertheless, the team has to distinguish between incrementally deliverable and system-wide non-functional requirements. The latter's will be developed in an incremental fashion as well, but their quality criteria can not be easily validated and demonstrated in an incremental process. Now if we bring in the habit of frequent refactoring, then all bets are off due to the potentially excessive, system-wide regression.
In summary, instead of flirting with the idea of tinkering, mixing and matching agile and non-agile practices, I would recommend simply avoiding "extremely agile" approaches for the problem area referred in the problem statement.
Tim Mackinnon:
I would definitely use agile methods for applications that can be built quickly. For safety critical systems I would support the use of more research into this area. I see a lot of examples of plan driven methodologies not delivering on their promise either.
Don Wells:
This would not change anything. There are plenty of throw-away applications being developed without adhering to a rigorous methodology now.
Bill Wood:
I believe that the customer needs to address the costs of failure in prioritizing their wants. XP in particular emphasizes the developer estimate the costs of implementing functionality, but an emphasis also needs to be placed upon the customer to estimate the costs of not having that functionality. If the costs of failure are huge, that functionality can be extensively tested without preventing future changes to the design or code.