We study a stylized multiple testing problem where the test statistics are independent and assumed to have the same distribution under their respective null hypotheses. We first show that, in the normal means model where the test statistics are normal Z-scores, the well-known method of (Benjamini and Hochberg, 1995) is optimal in some asymptotic sense. We then show that this is also the case of a recent distribution-free method proposed by Foygel-Barber and Candes (2015). The method is distribution-free in the sense that it is agnostic to the null distribution — it only requires that the null distribution be symmetric. We extend these optimality results to other location models with a base distribution having fast-decaying tails.
This is a joint research with Shiyun Chen. The corresponding paper is available online at https://arxiv.org/abs/1604.07520