Chapter 3: Measure Twice, Cut Once: Upstream Prerequisites

• A focus on prerequisites can reduce costs regardless of whether you use an iterative or sequential approach;
• Iterative approaches are usually a better option for many reasons, but an iterative approach that ignores prerequisites can end up costing significantly more than a sequential project that pays close attention to prerequisites;
• One realistic approach is to plan to specify about 80 percent of the requirements up front, allocate time for additional requirements to be specified later, and then practice systematic change control to accept only the most valuable new requirements as the project progresses;
• Another alternative is to specify only the most important 20 percent of the requirements up front and plan to develop the rest of the software in small increments, specifying additional requirements and designs as you go;
• You might choose a more sequential (up-front) approach when:
  • The requirements are fairly stable;
  • The design is straightforward and fairly well understood;
  • The development team is familiar with the applications area;
  • The project contains little risk;
  • Long-term predictability is important;
  • The cost of changing requirements, design, and code downstream is likely to be high.
• You might choose a more iterative (as-you-go) approach when:
  • The requirements are not well understood or you expect them to be unstable for other reasons;
  • The design is complex, challenging, or both;
  • The development team is unfamiliar with the applications area;
  • The project contains a lot of risk;
  • Long-term predictability is not important;
  • The cost of changing requirements, design, and code downstream is likely to be low.

3.3 Problem Definition Prerequisite

• first prerequisite before beginning construction is a clear statement of the problem that the system is supposed to solve. This is sometimes called “product vision”, “mission statement”, and “product definition”. But it really is a  “problem definition”;
• a problem definition defines what the problem is without any reference to possible solutions. It’s a simple statement, maybe one or two pages, and it should sound like a problem. The statement “We can’t keep up with orders for the X” sounds like a problem and is a good problem definition. The statement “We need to optimize our automated data-entry system to keep up with orders for the X” is a poor problem definition. It doesn’t sound like a problem; it sounds like a solution;
• a problem definition comes before detailed requirements work, which is a more in depth investigation of the problem;
• the programming process should be composed on the following layers:
  • problem definition;
  • requirements;
  • architecture;
  • construction;
  • system testing;
  • future improvements.
• the problem definition should be in user language and the problem should be described from a user’s point of view. Avoid stating it in technical computer terms;
• the exception to this rule applies when the problem is with the computer: compile times are too slow or the programming tools are buggy. In this case, it’s appropriate to state the problem in computer/programmer terms;
• without a good problem definition you might put effort into solving the wrong problem. Be sure you know what you’re aiming for;
• the penalty for failing to define the problem is that you can waste a lot of time solving the wrong problem. This is a double barreled penalty because you also don’t solve the right problem.

3.4 Requirements Prerequisite

• requirements describe in detail what a software system is supposed to do;
• requirements are the first step towards a solution;
• requirements are also known as: requirements development, requirements analysis, analysis, requirements definition, software requirements, specification, functional spec, spec;

Why have official requirements?

• ensure that the user rather than the programmer drives the system’s functionality. If requirements are explicit, the user can review and agree to them, otherwise the programmer usually ends up making requirements decisions during programming;
• explicit requirements keep you from guessing what the user wants;
• explicit requirements help avoid arguments between programmers. When disagreeing with another programmer about what is the program supposed to do, check the requirements;
• paying attention to requirements helps to minimize changes to a system after development begins. If you find a coding error during coding, you change a few lines of code and work goes on. If you find a requirements error during coding you have to alter the design to meet the changed requirement. Part of the old design needs to be thrown so design will take longer. Some code might get discarded and with it a few test cases. New code and test cases must be added. Unaffected code will need to be retested to ensure everything works;
• without good requirements, you can have the right general problem but miss the mark on specific aspects of the problem;
• specifying requirements adequately is a key to project success, perhaps even more important than effective construction techniques;

The Myth of Stable Requirements

• stable requirements are the holy grail of software development, but unfortunately it does not exist;
• on a typical project, the customer can’t reliably describe what is needed before the code is written. The more you work with a project, the better you understand it so the more the customer works with it the better he understands his needs and this is a major source of requirements changes;
• usually a project experiences about 25% change in requirements during development. This accounts for 70-85% of the rework on a typical project;
• requirements WILL change. The best thing to do is to minimize the impact of requirement changes;

Handling Requirements Changes During Construction

• if requirements aren’t good enough, stop and make them right before you proceed;
• make sure everybody knows the cost of requirement changes. When clients get excited about a feature they tend to get over all meetings / discussions / requirements. In that case, a good way to handle the situation would be to point out that it’s not in the requirements document and you will schedule a meeting and cost estimate to decide whether they want it now or later. The schedule and cost keywords will usually transform most must haves into nice to haves;
• if the organization is not sensitive to the importance of doing requirements first, point out that the changes at requirements time are much cheaper than changes later;
• when the client’s excitement persists, consider establishing a formal change-control board to review such proposed changes. This way you can adapt to their needs in a controlled way;
• use evolutionary prototyping approach to explore a system’s requirements before you send your forces in to build it. Build a little, get a little feedback, adjust the design, make a few changes, build a little more etc. The key is using short development cycles so that you can respond to your users quickly;
• dump the project if the requirements are especially bad or volatile and none of the above suggestions work. If you can’t drop it, think about how it would be like to cancel it. Think how much worse it would have to get for you to dump it and check the difference between the two cases;
• read requirements checklist.

Architecture Prerequisite

• software architecture is the high-level part of software design, the frame that holds the more detailed parts of the design;
• architecture is also knows as: system architecture, high-level design, top-level design;
• usually the architecture is described in a single document referred to as the architecture specification or top-level design;
• some people make a distinction between architecture an high-level design – architecture refers to design constraints that apply system-wide, whereas high-level design refers to design constraints that apply at the subsystem or multiple-class level, but not necessarily system wide;
• architecture should be a prerequisite because the quality of the architecture determines the conceptual integrity of the system. That in turn determines the ultimate quality of the system;
• a well thought-out architecture provides the structure needed to maintain a system’s conceptual integrity from the top levels down the bottom. It provides guidance to programmers—at a level of detail appropriate to the skills of the programmers and to the job at hand. It partitions the work so that multiple developers or multiple development teams can work independently;
• good architecture makes construction easy. Bad architecture makes construction almost impossible;
• without good software architecture, you may have the right problem but the wrong solution. It may be impossible to have a successful construction;
• like requirements changes, architectural changes are expensive to make during construction or later. So the earlier you can identify changes, the better.

Typical Architecture Components

Program Organization

• a system architecture first needs an overview that describes the system in broad terms. Without this overview, you’ll have a hard time building a coherent picture from a thousand details or even a dozen individual classes;
• architecture should provide evidence that alternatives to the final organization were considered and find the reasons the organization used was chosen over the alternatives. By describing organizational alternatives, the architecture provides the rationale for system organization and shows that each class has been carefully considered;
• design rationale is at least as important for maintenance as the design itself;
• the architecture should define the major building blocks in a program. Depending on the size of the program, each building block might be a simple class or a subsystem of many classes;
• every feature listed in the requirements should be covered by at least one building block;
• if a function is claimed by two or more building blocks, their claims should cooperate, not conflict;
• a building block should have one area of responsibility and this responsibility should be well defined;
• each building block should know as little as possible about other building blocks and their area of responsibility;
• the communication rules for each building block should be well defined;
  the architecture should describe which other building blocks the building block can use directly, which it can use indirectly and which it shouldn’t use at all;

Major Classes

• the architecture should specify the major classes to be used;
• the architecture should identify the responsibilities of each major class and how the class will interact with other classes;
• if the system is large enough, it should describe how classes are organized;
  the architecture should describe other class designs that were considered and give reasons for preferring the organization that was chosen;
• the architecture doesn’t need to specify every class in the system, aim for the 80/20 rule – specify the 20 percent of the classes that make up 80 percent of the system’s behavior;

Data Design

• the architecture should describe major files and table designs to be used;
• it should describe alternatives that were considered and justify the choices that were made. During construction, such information gives you insight into the minds of the architects. During maintenance, the same insight is an invaluable aid. Without it you’re watching a foreign movie with no subtitles;
• data should normally be accessed directly by only one of subsystem or classes, except through access classes or routines that allow access to the data in controlled and abstract ways;
• the architecture should specify high-level organization and contents of databases used and explain why a single database is preferable to multiple databases and vice versa. Identify possible interactions with other programs that access the same data, explain what views have been created on the data and so on;

Business Rules

• if the architecture depends on specific business rules, it should identify them and describe the impact the rules have on the system’s design;

User Interface Design

• sometimes the user interface is specified at requirements time. If it’s not, it should be specified in the software architecture;
• the architecture should specify major elements of the web page formats, GUIs, command line interfaces etc;
• careful architecture of the user interface makes the difference between a well-liked program and one that’s never used;
• the architecture should be modularized so that a new UI can be substituted without affecting the business rules and output parts of the program. For example, the architecture should make it fairly easy to remove a group of interactive interface classes and plug in a group of command line classes. This is useful for testing at the unit or subsystem level.

Input/Output

• architecture should specify a look-ahead, look-behind or just in time reading scheme;
• it should also describe the level at which I/O errors are detected: at field, record, stream or file level.

Resource Management

• the architecture should describe a plan for managing limited resources such as database connections, threads and handles;
• the architecture should estimate the resources used for nominal and extreme cases;

Security

• architecture should describe the approach to design-level and code-level security. If a threat model has not previously been built, it should be built at architecture time;
• coding guidelines should be developed with security implications in mind, including approaches to handling buffers; rules for handling untrusted data (data input from users, cookies, configuration data, other external interfaces); encryption; level of detail contained in error messages; protecting secret data that’s in memory; and other issues;

Performance

• performance goals should be specified in the requirements;
• performance goals should include both speed and memory use;
• the architecture should provide estimates and explain why the architects believe the goals are achievable;
• if certain areas are at risk of failing the use of specific algorithms or data types, the architecture should say so;
• if certain areas require the use of specific algorithms or data types to meet their performance goals, the architecture should say so;
• the architecture can also include space and time budgets for each class or object;

Scalability

• is the ability of a system to grow to meet future demands;
• the architecture should describe how the system will address growth in number of users, number of servers, number of network nodes, database size, transaction volume and so on;
• if the system is not expected to grow and scalability is not an issue, the architecture should make the assumption explicit;

Interoperability

• if the system is expected to share data or resources with other software or hardware, the architecture should describe how that will be accomplished;

Internationalization/Localization

• internationalization is a technical activity of preparing a program to support multiple locales;
• internationalization is often known as I18N because the first and last characters in internationalization are I and N and because there are 18 letters in the middle of the word;
• localization is known as L10N for the same reason and is the activity of translating a program to support a specific local language;
• most interactive systems contain dozens or hundreds of prompts, status displays, help messages, error messages and so on;
• the architecture should show that the typical string and character set issues have been considered including character set used, kinds of strings used, maintaining the strings without changing code and translating the strings into foreign languages with minimal impact on the code and the UI;
• the architecture can decide to use strings in line in code where they’re needed, keep strings in a class and reference them through the class interface or store the strings in a resource file. Either way, the architecture should explain which option has been chosen and why;

Error Processing

• some people have estimated that as much as 90% of a program’s code is written for exceptional, error-processing cases  or housekeeping, implying that only 10% is written for the nominal cases. Therefore, a strategy for handling the errors consistently should be spelled out in the architecture;
• error handling is often treated as a coding convention level issue, if it’s treated at all. But because it has system wide implications, it is best treated at the architectural level;
• here are some questions to consider:
  • is error processing corrective or merely detective? If corrective, the program can attempt to recover from errors. If it’s merely detective, the program can continue processing as if nothing had happened, or it can quit. In either case, it should notify the user that it detected an error;
  • Is error detection active or passive? The system can actively anticipate errors—for example, by checking user input for validity—or it can passively respond to them only when it can’t avoid them—for example, when a combination of user input produces a numeric overflow. It can clear the way or clean up the mess. Again, in either case, the choice has user-interface implications;
  • How does the program propagate errors? Once it detects an error, it can immediately discard the data that caused the error, it can treat the error as an error and enter an error-processing state, or it can wait until all processing is complete and notify the user that errors were detected (somewhere);
  • What are the conventions for handling error messages? If the architecture doesn’t specify a single, consistent strategy, the user interface will appear to be a confusing macaroni-and-dried-bean collage of different interfaces in different parts of the program. To avoid such an appearance, the architecture should establish conventions for error messages;
  • Inside the program, at what level are errors handled? You can handle them at the point of detection, pass them off to an error-handling class, or pass them up the call chain;
  • What is the level of responsibility of each class for validating its input data? Is each class responsible for validating its own data, or is there a group of classes responsible for validating the system’s data? Can classes at any level assume that the data they’re receiving is clean?
  • Do you want to use your environment’s built-in exception handling mechanism, or build your own? The fact that an environment has a particular error handling approach doesn’t mean that it’s the best approach for your requirements;

Fault Tolerance

• architecture should also indicate the kind of fault tolerance expected. Fault tolerance is a collection of techniques that increase a system’s reliability by detecting errors, recovering from them if possible, and containing their bad effects if not;

Architectural Feasibility

• the architecture should demonstrate that the system is technically feasible;
• if infeasibility in any area could render the project unworkable, the architecture should indicate how those issues have been investigated—through proof-of-concept prototypes, research, or other means;
• these risks should be resolved before full-scale construction begins;

Over Engineering

• robustness is the ability of a system to continue to run after it detects an error;
  often an architecture specifies a more robust system than that specified by the requirements. One reason is that system composed of many parts that are minimally robust might be less robust than is required overall;
• in software, the chain isn’t as strong as its weakest link, it’s as weak as all the weak links multiplied together;
• the architecture should clearly indicate whether programmers should err on the side of over engineering or on the side of doing the simplest thing that works;
  many programmers over engineer their classes automatically out of a sense of professional pride;
• by setting expectations explicitly in the architecture, you can avoid the phenomenon in which some classes are exceptionally robust and others are barely adequate;

Buy vs Build Decisions

• the most radical solution to building software is not to build it at all—to buy it instead;
• you can buy GUI controls, database managers, image processors, graphics and charting components, Internet communications components, security and encryption components, spreadsheet tools, text processing tools—the list is nearly endless;
• If the architecture isn’t using off-the-shelf components, it should explain the ways in which it expects custom-built components to surpass ready-made libraries and components;

Reuse Decisions

• if the plan calls for using pre-existing software, the architecture should explain how the reused software will be made to conform to the other architectural goals—if it will be made to conform;

Change Strategy

• building a software product is a learning process for both the programmers and the users, the product is likely to change throughout its development. The changes can be new capabilities likely to result from planned enhancements, or they can be capabilities that didn’t make it into the first version of the system. Consequently, one of the major challenges facing a software architect is making the architecture flexible enough to accommodate likely changes;
• the architecture should clearly describe a strategy for handling changes. The architecture should show that possible enhancements have been considered and that the enhancements most likely are also the easiest to implement;

Global Architectural Quality

• a good architecture specification is characterized by discussions of the classes in the system, of the information that’s hidden in each class, and of the rationales for including and excluding all possible design alternatives;
• a good architecture should fit the problem. When you look at the architecture, you should be pleased by how natural and easy the solution seems. It shouldn’t look as if the problem and the architecture have been forced together with duct tape. You might know of ways in which the architecture was changed during its development. Each change should fit in cleanly with the overall concept;
• the architecture’s objectives should be clearly stated. A design for a system with a primary goal of modifiability will be different from one with a goal of uncompromised performance, even if both systems have the same function;
• the architecture should describe the motivations for all major decisions. Be wary of “we’ve always done it that way” justifications which are not good enough if arguments are not brought to the table. The reason why it’s always done in a certain way might be an unexpected and unrelated one;
• good software architecture is largely machine and language independent. By being as independent of the environment as possible, you avoid the temptation to over-architect the system or to do a job that you can do better during construction;
• the architecture should tread the line between under-specifying and over-specifying the system. No part of the architecture should receive more attention than it deserves, or be over-designed;
• the architecture should address all requirements without gold-plating (without containing elements that are not required);
• the architecture should explicitly identify risky areas. It should explain why they’re risky and what steps have been taken to minimize the risk;
• finally, you shouldn’t feel uncomfortable about any parts of the architecture. It shouldn’t contain anything just to please the boss;
• the architecture shouldn’t contain anything that’s hard for you to understand. You’re the one who’ll implement it; if it doesn’t make sense to you, how can you implement it?
• be sure to check your architecture against the architecture check list to make sure you haven’t forgot anything.

Amount of Time to Spend on Upstream Prerequisites

• the amount of time to spend on problem definition, requirements, and software architecture varies according to the needs of your project. Generally, a well-run project devotes about 10 to 20 percent of its effort and about 20 to 30 percent of its schedule to requirements, architecture, and up-front planning. These figures don’t include time for detailed design that’s part of construction;
• if requirements are unstable and you’re working on a large, formal project, you’ll probably have to work with a requirements analyst to resolve requirements problems that are identified early in construction. Allow time to consult with the requirements analyst and for the requirements analyst to revise the requirements before you’ll have a workable version of the requirements;
• if requirements are unstable and you’re working on a small, informal project, allow time for defining the requirements well enough that their volatility will have a minimal impact on construction;
• if the requirements are unstable on any project—formal or informal—treat requirements work as its own project. Estimate the time for the rest of the project after you’ve finished the requirements. This is a sensible approach since no one can reasonably expect you to estimate your schedule before you know what you’re building;
• when allocating time for software architecture, use an approach similar to the one for requirements development. If the software is a kind that you haven’t worked with before, allow more time for the uncertainty of designing in a new area. Ensure that the time you need to create a good architecture won’t take away from the time you need for good work in other areas. If necessary, plan the architecture work as a separate project too;
• check out upstream prerequisites check list;
• check the additional resources (books) on requirements, architecture and development.

Key Points:

• the overarching goal of preparing for construction is risk reduction. Be sure your preparation activities are reducing risks, not increasing them;
• if you want to develop high-quality software, attention to quality must be part of the software-development process from the beginning to the end. Attention to quality at the beginning has a greater influence on product quality than attention at the end;
• part of a programmer’s job is to educate bosses and coworkers about the software-development process, including the importance of adequate preparation before programming begins;
• the kind of project you’re working significantly affects construction prerequisites—many projects should be highly iterative, and some should be more sequential.
  if a good problem definition hasn’t been specified, you might be solving the wrong problem during construction;
• if a good requirements work hasn’t been done, you might have missed important details of the problem. Requirements changes cost 20 to 100 times as much in the stages following construction as they do earlier, so be sure the requirements are right before you start programming;
• if a good architectural design hasn’t been done, you might be solving the right problem the wrong way during construction. The cost of architectural changes increases as more code is written for the wrong architecture, so be sure the architecture is right too;
• understand what approach has been taken to the construction prerequisites on your project and choose your construction approach accordingly.