Confidence Intervals

So far in the book we have focused on estimating a single value statistic. For example, let’s say we want to estimate the mean weight of adults in the NHANES dataset, so we take a sample from the dataset and estimate the mean. In this sample, the mean weight was 79.92 kilograms. We refer to this as a point estimate since it provides us with a single number to describe our estimate of the population parameter. However, we know from our earlier discussion of sampling error that there is some uncertainty about this estimate, which is described by the standard error.

You should also remember that the standard error is determined by two components:

1. the population standard deviation (which is the numerator), and
2. the square root of the sample size (which is in the denominator).

The population standard deviation is generally unknown and a fixed parameter that is not under our control, whereas the sample size is under our control. Thus, we can decrease our uncertainty about the estimate by increasing our sample size – up to the limit of the entire population size (which is very rare and if you get to this point, then there is no uncertainly at all given that you can just calculate the population parameter).

We would often like to have a way to more directly describe our uncertainty about a statistical estimate, which we can accomplish using confidence intervals. Most people are familiar with confidence intervals through the idea of a “margin of error” for political polls. These polls usually try to provide an answer that is accurate within +/- 3 per cent. For example, when a candidate is estimated to win an election by 9 percentage points with a margin of error of 3, the percentage by which they will win is estimated to fall within 6-12 percentage points.

In statistics, we refer to this kind of range of values as a confidence interval – the wider the confidence intervals are, the more uncertain we are about our parameter estimate.

Confidence intervals are notoriously confusing, primarily because they don’t mean what we might intuitively think they mean. If I tell you that I have computed a “95% confidence interval” for my statistic, then it would seem natural to think that we can have 95% confidence that the true parameter value falls within this interval. However, as we will see throughout the book, concepts in statistics often don’t mean what we think they should mean. In the case of confidence intervals, we can’t interpret them in this way because the population parameter has a fixed value – it either is or isn’t in the interval – and there is no way for us to know. Therefore, it doesn’t make sense to talk about the probability of that occurring. Jerzy Neyman, the inventor of the confidence interval, said, “the parameter is an unknown constant and no probability statement concerning its value may be made” (Neyman, 1937).^[1]

Instead, we have to view the confidence interval procedure from the same standpoint that we viewed hypothesis testing – as a procedure that in the long run will allow us to make correct statements with a particular probability. Thus, the proper interpretation of the 95% confidence interval is that – it is an interval that will contain the true population mean 95% of the time and for the intervals to not include the true population mean about 5% of the time.

Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude – not just, does a treatment affect people, but how much does it affect them. (Gene Glass, cited in Sullivan & Feinn 2012).^[5]

In the previous section, we discussed the idea that statistical significance may not necessarily reflect practical significance. In order to discuss practical significance, we need a standard way to describe the size of an effect in terms of the actual data, which we refer to as an effect size. In this section, we will introduce the concept and discuss various ways that effect sizes can be calculated.

An effect size is a standardised measurement that compares the size of some statistical effect to a reference quantity, such as the variability of the statistic. In some fields of science and engineering, this idea is referred to as a “signal to noise ratio”. There are many different ways that the effect size can be quantified, which depend on the nature of the data.

Cohen’s d

One of the most common measures of effect size is known as Cohen’s d, named after the statistician Jacob Cohen (who is most famous for his 1994 paper titled “The Earth is Round (p < .05)”)^[6]. It is used to quantify the difference between two means, in terms of their standard deviation:

where M₁ and M₂ are the means of the two groups, and S_pooled is the pooled standard deviation (which is a combination of the standard deviations for the two samples, weighted by their sample sizes). Note that this is very similar in spirit to the t statistic – the main difference is that the denominator in the t statistic is based on the standard error of the mean, whereas the denominator in Cohen’s d is based on the standard deviation of the data. This means that while the t statistic will grow as the sample size gets larger, the value of Cohen’s d will remain the same.

Table 6.6.1. Interpretation of Cohen’s D 

D	Interpretation
0.0 – 0.2	negligible
0.2 – 0.5	small
0.5 – 0.8	medium
> 0.8	large

Table 6.6.1 shows the commonly used interpretation for Cohen’s d. It can be useful to look at some commonly understood effects to help understand these interpretations. For example, the effect size for gender differences in adults’ height is d = 2.01. According to the table above, this is a very large effect. We can also see this by looking at the distributions of male and female heights in a sample from the NHANES dataset. Figure 6.6.4 shows that the two distributions are quite well separated, though still overlapping, highlighting the fact that even when there is a very large effect size for the difference between two groups, there will be individuals from each group that are more like the other group.

It is also worth noting that we rarely encounter effects of this magnitude in science, in part because they are such obvious effects that we don’t need scientific research to find them. Very large reported effects in scientific research often reflect the use of questionable research practices rather than truly huge effects in nature. It is also worth noting that even for such a huge effect, the two distributions still overlap – there will be some females who are taller than the average male, and vice versa. For most interesting scientific effects, the degree of overlap will be much greater, so we shouldn’t immediately jump to strong conclusions about individuals from different populations based on even a large effect size.

Pearson’s r

Pearson’s r, also known as the correlation coefficient, is a measure of the strength of the linear relationship between two continuous variables. We will discuss correlation in much more detail in the later chapters, therefore, we simply introduce r as a way to quantify the relation between two variables.

Pearson’s r is a measure that varies from -1 to 1, where a value of 1 represents a perfect positive relationship between the variables, 0 represents no relationship, and -1 represents a perfect negative relationship. Figure 6.6.5 shows examples of various levels of correlation using randomly generated data.

Odds ratio

In our earlier discussion of probability, we discussed the concept of odds – that is, the relative likelihood of some event happening versus not happening:

Previously, we discussed the ideas of false positives (aka Type I error) and false negatives (aka Type II error). People often focus heavily on Type I error, because making a false positive claim is generally viewed as a very bad thing; for example, the now discredited claims by Wakefield (1999)^[7] that autism was associated with vaccination led to anti-vaccine sentiment that has resulted in substantial increases in childhood diseases such as measles. Similarly, we don’t want to claim that a drug cures a disease if it really doesn’t. That’s why the tolerance for Type I errors is generally set fairly low, usually at $=0.05$. But what about Type II errors?

The concept of statistical power is the complement of Type II error – that is, it is the likelihood of finding a positive result given that it exists: power = 1-

In addition to specifying the acceptable levels of Type I and Type II errors, we also have to describe a specific alternative hypothesis – that is, what is the size of the effect that we wish to detect? Because will differ depending on the size of the effect we are trying to detect.

There are three factors that can affect statistical power:

• sample size: Larger samples provide greater statistical power
• effect size: A given design will always have greater power to find a large effect than a small effect (because finding large effects is easier)
• type I error rate: There is a relationship between Type I error and power such that (all else being equal) decreasing Type I error will also decrease power.

We can see this through simulation. First let’s simulate a single experiment, in which we compare the means of two groups using a standard t-test. We will vary the size of the effect (specified in terms of Cohen’s d), the Type I error rate, and the sample size, and for each of these we will examine how the proportion of significant results (i.e. power) is affected. Figure 6.6.6 shows an example of how power changes as a function of these factors.

There are at least two important reasons to care about statistical power. First, if you are a researcher, you probably don’t want to spend your time doing futile experiments. Running an underpowered study is essentially futile because it means that there is a very low likelihood that one will find an effect, even if it exists. Second, it turns out that any positive findings that come from an underpowered study are more likely to be false compared to a well-powered study.