Good Experimental Design Pays Dividends Continuously
Judicious selection of a logical sequence of designs leads to a model that pays for itself
 

The journalist David Brinkley once said “A successful man is one who can lay a firm foundation with the bricks others have thrown at him.” In science and engineering, a good experimental design is the key to a successful product, as well as understanding a process. A good design creates a useful mathematical model that can be used to understand and optimize a process. Once a team of subject matter experts is assembled, the process of experimental design can begin.

The first step should always be a statement of the objective, such as the design of a product or the improvement of a process. This leads naturally to the identification of the responses that need to be measured, such as the concentration of a known controlled material. The goal for each response should be defined at the outset, whether it should be minimized, maximized or achieve a target value. If there is a known range or acceptable specifications for the response, it should be identified. If there are responses of varying degrees of importance, they can be weighted to give more importance to the most critical. The procedure of response measurement should be identified and assessed for accuracy and precision before the experiment is conducted.

The input of a team is often needed to identify the factors that are considered to affect the response. If the experiment is early in the design of a process, an effort should be made to cast a wide net, but not too wide, since an experiment with too many factors often does not lead to a useful analytical outcome. If the experiment is at an improvement stage, the factors are often a manageably small set.

The type of factor should be declared — whether it is categorical, such as catalyst A or B; or continuous, such as pH. Additional factor types also may exist, such as
• those used in mixtures in the paint industry
• covariates which may confound or interact with other variables, such as the diet of patients in weight loss clinics
• constant factors fixed at a particular value, such as the volume of a reaction
• noise factors, which can be measured but not controlled, such as the humidity in a large room

The values, or levels, of the factors should be set wide enough that the effect of the factor on the response is felt but not so wide as to be impossible, unsafe or too costly. The difficulty of changing a factor needs to be identified, since this can make the execution of the design manageable by modification of the order of the runs (i.e. combinations of factor levels). Similarly, any combinations of factor levels that are impossible, unsafe or too costly need to be identified.

In order to select the design model, the objective should be reviewed. For early design stage projects, where the relationship between factors and responses is not well understood and there are a large number of factors, typically a screening design is used where the main effects (the effect of a factor uncontaminated by another factor) are of primary concern. For process improvement projects, where the relationship between factors and responses is understood and there are few key factors, typically a response surface design is used, where main effects and interactions between effects are considered.

An interaction occurs when the effect of one factor on a response depends on the level of another factor. Non-linear relationships between factors and responses also are considered in response surface designs by the inclusion of a center point, an intermediate value between the levels of a factor, to allow the calculation of curvature in the relationship between a factor and its response. Replication of runs (a particular value for each of the factors) also is important to allow the calculation of pure error, the portion of the total error that can not be explained regardless of which model is used, and the lack of fit error, which is calculated as the difference between total error and pure error.

If the lack of fit is significantly greater than the pure error, this is an indication that there is the wrong form of one of the factors. The run order also is important, since it should be randomized as much as possible to avoid any variable not present in the model from being confounded with a variable in the model. The total number of runs is usually limited by laboratory capacity, but also determined by how many main effects and interactions can be calculated.

The general design flow is summarized in Figure 1. At this point, the reader may be wondering exactly which design is to be selected based on the decisions they have made up to this point. There are two possible approaches based on the user’s knowledge of experimental designs and their inclinations. The reader can use a diagram, such as the one shown in Figure 2 on page 19, to identify the particular design, such as a Central Composite and then select this design in the design of experiments (DOE) menu in an application, such as Minitab. Another approach is to have the user’s selections used by a software package, such as JMP, to automatically select the appropriate design using the Custom Design menu, although the user does have the option to use one of the more specific DOE menus such as Screening Design.

Once the design is outlined, it can be executed and the outcome analyzed by fitting a model. The outcome of the analysis will often lead to further experiments that augment the original design and confirm the understanding of existing regions in the design space. In other words, screening design experiments are usually followed by response surface modeling experiments. The judicious selection of a logical sequence of good experimental designs leads to a useful model that can produce an optimized process for a cost-effective product that will pay for itself over its lifetime.

Mark Anawis is a Principal Scientist and ASQ Six Sigma Black Belt at Abbott. He may be reached at [email protected].