Effect size

An effect size describes how large the relationship is between two variables. This information is important in scientific research. Often it is useful to know not only whether an experiment had an effect, but also the size of any effects. Effect sizes are also helpful in practical situations, for the purpose of making decisions.

For example, if aliens were to land on earth, how long would it take for them to realise that, on average, males are taller than females? The answer relates to the effect size of the difference in height between men and women. The larger the effect size, the easier it is to see that men are taller. If the height difference were small, then it would take quite a while (and much sampling) to notice that men were, on average, taller than women

The concept of an effect size appears in everyday language. For example, a weight loss program may boast that it leads to an average weight loss of 30 pounds. In this case, 30 pounds is an indicator of the claimed effect size. Another example is that a tutoring program may claim that it raises school performance by one letter grade. This grade increase is the claimed effect size of the program.

In inferential statistics, an effect size is the size of a statistically significant difference. Effect sizes, along with N and critical alpha determine power in statistical hypothesis testing. In meta-analysis, effect sizes are used as a common measure which can be calculated for different studies and then combined into overall analyses.

Types of effect sizes

Pearson r correlation

Pearson's r correlation is one of the most widely used effect sizes. It can be used when the data are continuous or binary, thus the Pearson r is arguably the most versatile effect size. This was the first important effect size to be developed in statistics, and it was introduced by Karl Pearson. Pearson's r can vary in magnitude from -1.00 to 1.00, with -1.00 indicating a perfect negative relationship, 1.00 indicating a perfect positive relationship, and zero indicating no relationship between two variables.

Another often used measure of the size of the relationship between two variables is the square or r, often referred to as "r-squared" or the coefficient of determination. It is a measure of the proportion of variance shared by the two variables and varies from zero to 1.00.

Cohen's d

Cohen's d is the appropriate effect size measure to use in the context of a t-test on means. d is defined as the difference between two means divided by the pooled standard deviation for those means. Thus,

d = {\mathrm{mean}_1 - \mathrm{mean}_2 \over \sqrt{(\mathrm{SD}_1^2 + \mathrm{SD}_2^2) /2 \ }}

where mean_i and SD_i are the mean and standard deviation for group i, for i = 1, 2.

Different people offer different advice regarding how to interpret the resultant effect size, but the most accepted opinion is that of Cohen (1992) where 0.2 is indicative of a small effect, 0.5 a medium and 0.8 a large effect size.

So, in the example of aliens observing men and women's height, the data (from a UK representative sample of 1000 men and 1000 women) could be:

Men: Mean Height = 1754 mm; Standard Deviation = 70.00 mm
Women: Mean Height = 1620 mm; Standard Deviation = 64.90 mm

The effect size (using Cohen's d) would equal 1.99. This is very large and aliens should have no problem in detecting that there is a substantial height difference.

One point worth noting, though, is that in some cases it may be wise to use a pooled standard deviation while in other cases it makes more sense to use just one of the standard deviations (e.g., pre-treatment standard deviation in a therapeutic trial). Either way, note that sample size and unequal sample size does not play a part in the calculation - points noted by Hedges.

Hedges' ĝ

Hedges and Olkin (1985) noted that one could adjust effect size estimates by taking into account the sample size. The problem with Cohen's d is that the outcome is heavily influenced by the denominator in the equation. If one standard deviation is larger than the other then the denominator is weighted in that direction and the effect size is more conservative. However, surely it makes more sense to put stock in the larger sample size? Hedges' ĝ incorporates sample size by both computing a denominator which looks at the sample sizes of the respective standard deviations and also makes an adjustment to the overall effect size based on this sample size. The formula for Hedges' ĝ (as used by software such as the Effect Size Generator) is

$\hat{g} = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{(n_1 - 1) SD_1^2 + (n_2 - 1) SD_2^2}{(N_\mathrm{total} - 2)}}} \times \bigg(1-\frac{3}{4(n_1+n_2)-9}\bigg).$

Cohen's $f^{2}$

Cohen's

f^{2}

is the appropriate effect size measure to use in the context of an F-test for multiple correlation or multiple regression. The

f^{2}

effect size measure for multiple regression is defined as:

f^{2} = {R^{2} \over 1 - R^{2}}

where

R^{2}

is the squared multiple correlation.

The $f^{2}$ effect size measure for hierarchical multiple regression is defined as:

f^{2} = {(R^{2}_{AB} - R^{2}_A) \over 1 - R^{2}_{AB}}

where

R^{2}_A

is the variance accounted for by a set of one or more independent variables A, and

R^{2}_{AB}

is the combined variance accounted for by A and another set of one or more independent variables B.

By convention, $f^{2}$ effect sizes of 0.02, 0.15, and 0.35 are considered small, medium, and large, respectively (Cohen, 1988).

Odds ratio

The odds ratio is another useful effect size. It is appropriate when both variables are binary. For example, consider a study on spelling. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or more briefly 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. However, odds ratio statistics are on a different scale to Cohen's d. So, this '3' is not comparable to a Cohen's d of '3'.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum

Cohen, J. (1992). A power primer. Psychological Bulletin, 112 (1), 155-159.

Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Sage: Thousand Oaks, CA.

External links

Software

Free Effect Size Generator Software - PC & Mac Software
Free Odds Ratio Generator Software - PC Software
Free GPower Software - PC & Mac Software
Free Effect Size Calculator for Multiple Regression - Web Based
Free Effect Size Calculator for Hierarchical Multiple Regression - Web Based

Further Explanations

clinical research | Educational psychology | pharmaceutical industry | Hypothesis testing

Gourt Home::Gourt :: Effect size