Describe the use of a power curve and the tradeoff between T

Describe the use of a power curve and the tradeoff between Type I and II errors. When is Type I error more important to consider than Type II? Vice versa? What experimental considerations are determined by power analysis?

Solution

In many disciplines (including mine, Psychology) classical hypothesis testing is the usual method of analyzing research data. Typically we have a relatively small sample of data and we employ a .05 (alpha) criterion of significance, a combination which makes a Type II error much more probable than a Type I error. This is probably quite reasonable for much of the research that is done in my discipline (where the null hypothesis is usually that there is no relationship between two variables or sets of variables), but it might not always be reasonable. Rarely do we consider why the .05 criterion is used, and often we don\'t consider the effect of varying sample size. To stimulate thought on this matter, I suggest you imagine that you are testing an experimental drug that is supposed to reduce blood pressure, but is suspected of inducing cancer. You administer the drug to a sample of rodents. Since you know that the tumor rate in this strain is 10% among untreated animals, your null hypothesis (the one which includes an equals sign) is that the tumor rate in treated animals is less than or equal to 10%, that is, the drug is safe, it does not cause tumors. The alternative hypothesis is that the tumor rate in treated animals is more than 10%, that is, the drug is not safe. A Type I error is defined as rejecting a true null hypothesis (not being a believer in the utility of testing point null hypotheses, what I really mean here is rejecting a null hypothesis that is so close to true that for practical purposes it is true). In this example that amounts to concluding that the drug is not safe when in fact it is. A Type II error is defined as failing to reject a false null hypothesis -- here, concluding that the drug is safe when in fact it is not.

If you were a potential consumer of this new drug, which of these types of errors would you consider more serious? Your initial response might be that it is more serious to make the Type II error, to declare an unsafe drug as being safe. Having decided that the Type II error is more serious, one should consider techniques to decrease the probability of making such an error, beta. One way to decrease beta is to increase alpha. That is, one might be willing to trade an increased risk of a Type I error for a decreased risk of a Type II error. It is, however, possible to decrease beta without increasing alpha. Additional power (ability to detect the falsity of the null hypothesis, (1 - beta) may be obtained by using larger sample sizes, more efficient statistics, and/or by reducing \"error variance\" (any variance in the dependent variable not caused by the independent variable).

There are, however, several difficult to quantify factors that we have not considered so far in our evaluation of the relative seriousness of Type I and Type II errors. What if you are one of those persons for whom currently available drugs are not effective? Might that make you reconsider the relative seriousness of the two types of errors? Concluding that that drug is not safe when in fact it is (Type I error) may now seem the more serious error, since it denies you the opportunity to obtain a new drug which might save your life. Furthermore, even it the drug does \"significantly\" raise tumor rates, you might be willing to accept an increased risk of developing cancer in return for achieving effective control of your blood pressure. If we use methods that maximize power we run the risk of declaring as \"significant\" an increase in tumor rate which is quite small, too small to outweigh the potential benefits of the new drug (but large enough to attract the attention of attorneys who specialize in medical/pharmaceutical malpractice).

Now imagine that you are not a potential consumer of this drug but rather a stockholder in the pharmaceutical company whose primary concern is with the profits to be made in the short term. Concluding that the drug is unsafe, when it really is safe (Type I) now becomes an extremely serious error, one which could not only deny patients of a potentially useful medication but deny you your well deserved capital gains. You might also be less than enthusiastic about increasing power by gathering more data, since it costs money to gather more data and the increased power would make it more likely that you would detect an increase in tumor rate should one exist.

It might be useful to consider an economic analysis of the problem. You could attempt to quantify the likely costs associated with making the one or the other type of error, the costs of collecting additional data, and note how these costs change as you vary sample size and alpha, choosing the sample size and alpha which minimize the costs. But you and I might differ with respect to our quantification of the costs of Type I versus Type II errors, right?

Now imagine that we have decided that the drug is safe. To get approval to market the drug we must also show that it is effective. We test its effect on blood pressure. Our dependent variable is pre- treatment blood pressure minus post-treatment blood pressure. Positive scores indicate that the drug lowered blood pressure. The null hypothesis, with the equals sign, is that the mean decrease in blood pressure is less than or equal to zero, that is, the drug is not effective. The alternative hypothesis is that the mean decrease is greater than zero, the drug is effective. A Type I error is concluding that the drug is effective when in fact it is not. A Type II error is concluding that the drug is not effective when in fact it is. Most of my students initially opine that the Type I error is more serious in this example