Software development process

A software development process is a structure imposed on the development of a software product. Synonyms include software life cycle and software process. There are several models for such processes, each describing approaches to a variety of tasks or activities that take place during the process.

Processes and meta-processes

A growing body of software development organizations implement process methodologies. Many of them are in the defense industry, which in the U.S. requires a 'Rating' based on 'Process models' to obtain contracts. ISO 12207 is a standard for describing the method of selecting, implementing and monitoring a lifecycle for a project.

The Capability Maturity Model (CMM) is one of the leading models. Independent assessments can be used to grade organizations on how well they create software according to how they define and execute their processes. ISO 9000 describes standards for formally organizing processes with documentation.

ISO 15504, also known as Software Process Improvement Capability Determination (SPICE), is a "framework for the assessment of software processes". The software process life cycle is also gaining wide usage. This standard is aimed at setting out a clear model for process comparison. SPICE is used much like CMM and CMMI. It models processes to manage, control, guide and monitor software development. This model is then used to measure what a development organization or project team actually does during software development. This information is analyzed to identify weaknesses and drive improvement. It also identifies strengths that can be continued or integrated into common practice for that organization or team.

Six Sigma is a project management methodology that uses data and statistical analysis to measure and improve a company's operational performance. It works by identifying and eliminating "defects" in manufacturing and service-related processes. The maximum permissible defects are 3.4 per million opportunities. However Six Sigma is manufacturing-oriented, not software development-oriented and needs further research to even apply to software development.

Linus Torvalds, the project leader of the Linux kernel, made the following statement on the Linux kernel mailing list:

No major software project that has been successful in a general marketplace (as opposed to niches) has ever gone through those nice lifecycles they tell you about in CompSci classes.

Source: Google Groups archive of fa.linux.kernel discussion

Process Activities/Steps

Software Engineering processes are composed of many activities, notably the following. They are considered sequential steps in the Waterfall process, but other processes may rearrange or combine them in different ways.

Requirements Analysis : Extracting the requirements of a desired software product is the first task in creating it. While customers probably believe they know what the software is to do, it may require skill and experience in software engineering to recognize incomplete, ambiguous or contradictory requirements.
Specification : Specification is the task of precisely describing the software to be written, in a mathematically rigorous way. In practice, most successful specifications are written to understand and fine-tune applications that were already well-developed, although safety-critical software systems are often carefully specified prior to application development. Specifications are most important for external interfaces that must remain stable.
Software architecture : The architecture of a software system refers to an abstract representation of that system. Architecture is concerned with making sure the software system will meet the requirements of the product, as well as ensuring that future requirements can be addressed. The architecture step also addresses interfaces between the software system and other software products, as well as the underlying hardware or the host operating system.
Coding : Reducing a design to code may be the most obvious part of the software engineering job, but it is not necessarily the largest portion.
Testing : Testing of parts of software, especially where code by two different engineers must work together, falls to the software engineer.
Documentation : An important (and often overlooked) task is documenting the internal design of software for the purpose of future maintenance and enhancement. Documentation is most important for external interfaces.
Software Training and Support : A large percentage of software projects fail because the developers fail to realize that it doesn't matter how much time and planning a development team puts into creating software if nobody in an organization ends up using it. People are occasionally resistant to change and avoid venturing into an unfamiliar area, so as a part of the deployment phase, its very important to have training classes for the most enthusiastic software users (build excitement and confidence), shifting the training towards the neutral users intermixed with the avid supporters, and finally incorporate the rest of the organization into adopting the new software. Users will have lots of questions and software problems which leads to the next phase of software.
Maintenance : Maintaining and enhancing software to cope with newly discovered problems or new requirements can take far more time than the initial development of the software. Not only may it be necessary to add code that does not fit the original design but just determining how software works at some point after it is completed may require significant effort by a software engineer. About 2/3 of all software engineering work is maintenance, but this statistic can be misleading. A small part of that is fixing bugs. Most maintenance is extending systems to do new things, which in many ways can be considered new work. In comparison, about 2/3 of all civil engineering, architecture, and construction work is maintenance in a similar way.

Process Models

A decades-long goal has been to find repeatable, predictable processes or methodologies (software engineering) that improve productivity and quality. Some try to systematize or formalize the seemingly unruly task of writing software. Others apply project management techniques to writing software. Without project management, software projects can easily be delivered late or over budget. With large numbers of software projects not meeting their expectations in terms of functionality, cost, or delivery schedule, effective project management is proving difficult.

Waterfall processes

The best-known and oldest process is the waterfall model, where developers (roughly) follow these steps in order. They state requirements, analyze them, design a solution approach, architect a software framework for that solution, develop code, test (perhaps unit tests then system tests), deploy, and maintain. After each step is finished, the process proceeds to the next step, just as builders don't revise the foundation of a house after the framing has been erected. If iteration is not included in the planning, the process has no provision for correcting errors in early steps (for example, in the requirements), so the entire (expensive) engineering process may be executed to the end, resulting in unusable or unneeded software features.

In old style (CMM) processes, architecture and design preceded coding, usually by separate people in a separate process step.

Iterative processes

Iterative development prescribes the construction of initially small but ever larger portions of a software project to help all those involved to uncover important issues early before problems or faulty assumptions can lead to disaster. Iterative processes are preferred by commercial developers because it allows a potential of reaching the design goals of a customer who does not know how to define what he wants.

Agile software development processes are built on the foundation of iterative development. To that foundation they add a lighter, more people-centric viewpoint than traditional approaches. Agile processes use feedback, rather than planning, as their primary control mechanism. The feedback is driven by regular tests and releases of the evolving software.

Agile processes seem to be more efficient than older methodologies, using less programmer time to produce more functional, higher quality software, but have the drawback from a business perspective that they do not provide long-term planning capability. In essence, they say that they will provide the most bang for the buck, but won't say exactly when that bang will be.

Extreme Programming, XP, is the best-known agile process. In XP, the phases are carried out in extremely small (or "continuous") steps compared to the older, "batch" processes. The (intentionally incomplete) first pass through the steps might take a day or a week, rather than the months or years of each complete step in the Waterfall model. First, one writes automated tests, to provide concrete goals for development. Next is coding (by a pair of programmers), which is complete when all the tests pass, and the programmers can't think of any more tests that are needed. Design and architecture emerge out of refactoring, and come after coding. Design is done by the same people who do the coding. (Only the last feature - merging design and code - is common to all the other agile processes.) The incomplete but functional system is deployed or demonstrated for (some subset of) the users (at least one of which is on the development team). At this point, the practitioners start again on writing tests for the next most important part of the system.

While Iterative development approaches have their advantages, software architects are still faced with the challenge of creating a reliable foundation upon which to develop. Such a foundation often requires a fair amount of upfront analysis and prototyping to build a development model. The development model often relies upon specific design patterns and entity relationship diagrams (ERD). Without this upfront foundation, Iterative development can create long term challenges that are significant in terms of cost and quality.

Critics of iterative development approaches point out that these processes place what may be an unreasonable expectation upon the recipient of the software: that they must possess the skills and experience of a seasoned software developer. The approach can also be very expensive, akin to... "If you don't know what kind of house you want, let me build you one and see if you like it. If you don't, we'll tear it all down and start over." A large pile of building-materials, which are now scrap, can be the final result of such a lack of up-front discipline.

Formal methods

Formal methods are mathematical approaches to solving software (and hardware) problems at the requirements, specification and design levels. Examples of formal methods include the B-Method, Petri nets, RAISE and VDM. Various formal specification notations are available, such as the Z notation. More generally, automata theory can be used to build up and validate application behaviour by designing a system of finite state machines.

Finite state machine (FSM) based methodologies allow executable software specification and by-passing of conventional coding (see virtual finite state machine or event driven finite state machine).

Recent approaches try to merge the specification and code into one activity to ensure the specification and code match. While Agile methods propagate specification of all requirements in code, methods such as VFSM develop executable specifications, trying to avoid the coding activity entirely.