To p (< .005) or not to p (< .005), that's the question

As I am sure everyone already knows, there is a new discussion about p-values in the field of psychology/statistics. While I do not feel qualified to have a public opinion about this, I do like reading all the papers and replies to papers and comments on papers. It kind of feels like I am observing the new iteration of the “nature-nurture” debate unfold. The nerd in me is very excited. And so, I thought it would be nice to compile a reading list, so that others who are also excited can read everything out there about what is going on. I might update this as more things get published. Also, go to Twitter, because it is extremely entertaining right now (sadly, I could not get myself to screenshot every entertaining thread that’s been happening, but if you look up E-J Wagenmakers, Daniel Laken, Andrew Gelman, and the other authors mentioned below, it should be easy to find!).

The paper that started it all:

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., … & Cesarini, D. (2017). Redefine statistical significanceNature Human Behaviour, 1.

The papers that followed:

Greenland, S., & Amrhein, V. (2017). Remove, rather than redefine, statistical significanceNature Human Behaviour, 1.

Crane, H. (2017, November 19). Why “Redefining Statistical Significance” Will Not Improve Reproducibility and Could Make the Replication Crisis Worse. Retrieved from psyarxiv.com/bp2z4

Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., … Zwaan, R. A. (2017, September 18). Justify Your Alpha: A Response to “Redefine Statistical Significance”. Retrieved from psyarxiv.com/9s3y6

McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2017). Abandon statistical significance. arXiv preprint arXiv:1709.07588.

Trafimow, D., Amrhein, V., Areshenkoff, C. N., Barrera-Causil, C., Beh, E. J., Bilgiç, Y., … & Chaigneau, S. E. (2017). Manipulating the alpha level cannot cure significance testing – comments on “Redefine statistical significance”PeerJ Preprints.

Wicherts, J. M. (2017). The Weak Spots in Contemporary Science (and How to Fix Them)Animals7(12), 90.

Zenker, F., & Witte, E. H. (2017). From Discovery to Justification: Outline of an Ideal Research Program for Empirical PsychologyFrontiers in psychology8, 1847.

Some more informal replies:

E.J Wagenmakers (the paper that started it all) responds to points made in McShane et al. paper.

He also responds to Crane’s paper.

He also responds to Lakens et al.’s paper.

And he was on a panel at BITSS2017 with Daniel Lakens (see above) and Simine Vazire. If you want to watch the entire conference, you can do that here (the panel starts around the 30-minute mark on Day 1).

Basically, E.J. Wagenmakers has written a lot of responses to critiques on the paper that started it all. Here is the first, which mostly summarizes. The secondthird, fourth, fifth, sixth, and seventh go into more detail about why the paper that started it all is right, using examples (I’m still not sure).

Andrew Gelman and Christian Robert respond to E.J. Wagenmakers response.

And of course, E.J. Wagenmakers responds to their response.

In the meantime, Daniel Lakens has written a post on common misconceptions about p-values.

The protagonists (basically the big names in all the main publications, from all sides) also participated in a roundtable discussion hosted by the International Methods Colloquium. (thank you @lionbehrens)

Of course, this isn’t a new discussion:

Cohen, J. (1995). The Earth is Round (p < .05)American Psychologist, 50(12).

And its inevitable follow-up:

Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017). The earth is flat (p> 0.05): Significance thresholds and the crisis of unreplicable research. PeerJ Preprints.

If we want to get official about it:

The ASA’s Statement on p-Values: Context, Process, and Purpose.

Also, I really liked this paper targeting journalists:

Spotting Shady Statistics on The Open Notebook.

Just see this as a nice little reading list for the Winter Break. Oh? You already have 300 other things to work on during Winter Break? I have no idea how you feel. 😉

Advertisements

Podcasts I am listening to

In an attempt to make my procrastination a slight bit more useful than just zoning out to Netflix, I’ve found some podcasts that seem valuable and interesting to me (and probably a lot of other people). In an attempt to help others make their personal procrastination issues less severe, I want to share those podcasts here. I might add more as I discover them. As a note: I realize some of these are old news, but I still want to share them here for any other podcast newbies like me.

The Effort Report: Useful and entertaining advice on how to survive life as an academic (or “life in the academic trenches” as the creators describe it). Professors Elizabeth Matsui and Roger Peng (both from Johns Hopkins University) have been recording this podcast since July 2016. It already has 60 episodes, so there is plenty to listen to. Topics cover all aspects of academic life (e.g., how to write a fundable grant, the role of gender, where your salary comes from, finding and maintaining collaborations, and what your academic wardrobe should look like).

The Bayes Factor: A new podcast recorded by Alex Etz (UC Irvine) and J.P. de Ruiter (Tufts University) about the people behind Bayesian statistics and related methodological issues in psychological research. So far, this podcasts has 3 episodes, but they only started on November 3, 2017, so the number of episodes seems to grow at quite the pace. The hosts invite someone that has an opinion about a specific methodological issue (not necessarily Bayesian) and discuss/interview them.

The Black Goat: Hosted by Sanjay Srivastava (University of Oregon), Alexa Tullett (University of Alabama), and Simine Vazire (UC Davis). They are three psychologists who discuss different aspects of doing science. Also relatively new (I mean, I guess even the Effort Report is still somewhat new), this podcast started in March 2017. They have recorded 21 episodes so far on topics such as the role of teaching in a faculty member’s life, how to have a disagreement with other scientists, applying for an academic job opening, maintaining friendships outside of academia. The hosts discuss these topics by looking at their own personal experiences with these issues.

Not So Standard Deviations: A podcast about the latest in data science and data analysis in academia and industry. Created by Roger Peng (yes, the same as the co-host of The Effort Report… Where does he find the time to do all of his other work?) and Hilary Parker (who works at Stitch Fix – the company that sends you tailored-for-you clothing). The combination of someone from academics and someone from industry is very interesting and makes for good discussions. This podcast is focused more on data science as opposed to quantitative psychology/statistics. The hosts don’t just discuss the statistics, but also more technical issues such as private cloud servers and building data science products that have good user experience. This podcast started in September 2015 and currently has 49 episodes.

This is my list so far. When I succumb to more procrastination later, I might find some more!

UPDATED to add:

Everything Hertz: A podcast about methodology, scientific life, and (as they themselves say) bad language. Hosted by Dan Quintana (University of Oslo) and James Heathers (Northeastern University). This podcast is sort of a mix between the Effort Report and the Bayes Factor (though not really focused on Bayes). What I’m trying to say is that they talk about topics that have to do with professional development and the general research process. But then they also talk very specifically about how to interpret effect sizes and how we are misunderstanding p-values. They have already recorded 54 episodes in about a year, so there’s plenty to listen to. Daniel Lakens (Eindhoven University) is a frequent guest on the podcast, which leads to some interesting conversations.

The Startup Scientist: This podcast is also hosted by Dan Quintana. He seems to have recorded some episodes earlier this year, then he took a break but now he’s back. He talks about professional development issues that are important to any researcher in any field (think: building your online presence, picking a research project) These podcasts are like mini-podcasts, with about 6 minutes of material at a time. So if you’re just waiting for the bus or the start of a class, these would be perfect. I hope he keeps adding new episodes!

What am I doing?!

This is a question I often ask myself. I mean this both literally, as in “what am I doing at this exact moment? What should I be doing? What still needs to be done?”, and more holistically, as in “what are my goals as a Ph.D. student? Am I doing everything I can to further my academic career? Am I screwing it all up?”.

For the first type of question, I have lists. Endless lists of things that need to be done. Little pieces of the big pie called a Ph.D. For the second type of question, it is more complicated. I might consult my advisor, my fellow Ph.D. students, an unlucky barista at Starbucks…

I guess what I am trying to say here is that this is why I want to write here. I want to have a record of what I am doing, what I am thinking, what is going on in my life, in my career. If that sounds at all interesting to you, please follow along! If not, no hard feelings 😉 I’m just self-involved enough to admit that this blog is mostly just for me.