Link to Content:
Multiple Hypothesis Testing
Content Found Via:
Tags: hypothesis testing / statistics
“In recent years, there has been a lot of attention on hypothesis testing and so-called ‘p-hacking’, or misusing statistical methods to obtain more ‘significant’ results….
This post introduces some of the interesting phenomena that can occur when we are dealing with testing hypotheses. First, we consider an example of a single hypothesis test which gives great insight into the difference between significance and “being correct”. Next, we look at global testing, where we have many different hypotheses and we want to test whether all null hypotheses are true using a single test. We discuss two different tests, Fisher’s combination test and Bonferroni’s method, which lead to rather different results. We save the best till last, when we discuss what to do if we have many hypotheses and want to test each individually. We introduce the concepts of familywise error rate and false discovery rate, and explain the Benjamini-Hochberg procedure.
Also, this post is accompanied by an IPython notebook that demonstrates how these methods work in practice. We analyze free throw percentage data from the NBA to see whether there are players that perform better, or worse, playing at home versus away.”
Recommended Prerequisites: this post assumes basic knowledge about hypothesis testing, such as the difference between the null and alternative hypotheses, significance levels, Type I & II errors, p-values.
Go to Content: Multiple Hypothesis Testing